Discussion:
Starting from Scratch - What Would You Do?
(too old to reply)
Mark Jeffreys
2008-09-08 23:09:42 UTC
Permalink
Folks:

The recent thread "Things I've learned [Long]" got me thinking...

Sometime in the next few months, it looks like I'll have the chance to do
something I've been wanting to do for a long time, that being a major
rewrite of JobTrack, our main production database in Creative Services (i.
e., the in-house advertising & interactive agency). I started working on
this in 1992, when it was still quite small; someone in the department had
begun writing it in FileMaker but then realized that FileMaker wasn't up to
the task and started rewriting it in 4D. I was called in to assist and have
been working on it ever since, first as a part-time contractor, then
full-time, and in recent years as an employee. It was initially based on a
skeleton I had developed in version 3 when I was working at UCSF, which was
itself based on a version 2 database, which was in turn based on the first
DB I wrote in version 1 back in 1987-88. (These are all the US version
numbers, by the way.)

JobTrack has grown from a relatively simple tool that only recorded general
information about jobs to a mission-critical app that is used to track staff
hours, staff scheduling, interaction with other tools and interfaces on the
intranet, various accounting and billing functions and reports, management
overviews and on and on. Although I did a fairly extensive rewrite back in
1999-2000 (when we went to version 6, IIRC), there is still quite a lot of
code that dates back to the version 3 days, and maybe even some left from
version 2 here and there. What I wasn't able to do in that rewrite was to
make some of the larger structural changes that I would have liked to make
-- change some things that even back then I knew were undesirable, and now
know to be hampering my ability to maintain and extend JobTrack. Many of
these were due to either design decisions made early on, some even before I
came on board, or were due to the fact that my skill and experience in
design and programming just were not as developed back then. (As a random
but important example, using "meaningful" table keys -- man, that's caused
me a lot of grief!). Another consideration is getting rid of a lot of
unnecessary bloat, such as various denormalizations and summary tables that
were required for decent performance 10 years ago but now just complicate
matters.

So I now hope to have an opportunity not many of us get: to start
more-or-less from scratch, and rebuild JobTrack in a manner that will allow
me to spend less time tiptoeing through the shaky foundations and more time
actively extending and improving it. Now that I'm starting to delve
seriously into v11, with all the new options available therein, it seems
like the right time to start fresh. I've been thinking about how to approach
this, given all the paradigms and techniques I've encountered over the years
from the iNUG and other sources. I'd like to start off with a list of
principles -- what I should and shouldn't do -- and I'd like your advice as
to what that list should contain. Here are a handful that seem obvious:

- Unit testing/automated testing
- Strict modularization
- Layered architecture (3-tier? MVC? Other?)
- Function wrapping
- Accessor methods for global(ish) values
- Ubiquitous error handling
- Source control
- Custom data structures where appropriate
- Administrative interface *

I've been using some of these principles for years as I've expanded or
updated parts of the code base (modules, error handling, accessors). Others
I'm aware of as concepts but would like to have specific examples and advice
for implementation, especially unit/automated testing and layered
architecture. And I'm sure there are many important items I've left off this
initial list.

Note that I'm not trying to provoke arguments about the superiority of one
approach over another; I'm trying to compile a list of things I shouldn't
ignore without due consideration. I don't want to get some months into a
huge undertaking only to realize that I've designed or coded myself into a
corner because I missed something I should have done at the beginning.

That's it -- where would you start and what would you do if you had a chance
to do it all over again?

Thanks in advance,

-- MLJ


* I've always used the User environment for a lot of ad hoc reporting and
data massaging on the live (compiled) server, and it's gone away in v11.
Although that wouldn't matter nearly as much if I could set the standard
Query Editor in Application mode to remember what had been entered when last
used!
--
Mark L. Jeffreys
Senior Staff Technology Solutions
Creative Services
Charles Schwab & Co., Inc.
San Francisco CA

Warning: All e-mail sent to this address will be received by the Charles
Schwab & Co. corporate e-mail system and is subject to archival and review
by someone other than the recipient.

**********************************************************************
4D Server v11 SQL has arrived!
Buy it NOW at http://store.4ddepot.com

4th Dimension Internet Users Group (4D iNUG)
FAQ: http://www.4d.com/support/faqnug.html
Archive: http://dir.gmane.org/gmane.comp.lang.inug-4d.tech
Unsub: mailto:4D_Tech-Unsubscribe-d2/MUvgItPNWk0Htik3J/***@public.gmane.org
Post: mailto:4d_tech-jTdV3SBRT1BWk0Htik3J/***@public.gmane.org
Options: https://lists.4d.com/mailman/listinfo/4d_tech
**********************************************************************
David Dancy
2008-09-08 23:30:11 UTC
Permalink
Mark

Lucky you!

Here's what I'd do:

1. Don't throw out the old stuff. It's working!
2. Develop the new stuff in v11 in a converted database that has all
the old stuff (including tables) still in it. There are plenty of
tables to go around - you won't run out, I promise!
3. Develop the new stuff just-how-you-like-it, with your ideal
structures, normalised tables, well-named key fields, and so on,
_alongside the old stuff_. Then write data conversion methods to
convert the old data to your new design.
4. Abstract a Data Access Layer out of your existing code so that it
deals with Business Objects rather than records, if you haven't done
so already. Then write code to fill in the Business Objects from both
your old and new data structures and save them back. This will also
provide your customers with a transparent upgrade path, and it won't
require a massive export-import operation for them. Your new program
can read data from the old design and write it back to the new design
without the customer being any the wiser, especially if you make the
old tables invisible.
5. Use SQL where you can for data access, unless the 4D commands are
genuinely superior or you need the convenience of a Current Selection.
This will mean that you could make use of DB engines other than 4D if
necessary/convenient, and it will provide you with a means of
unlocking and re-using your Intellectual Property investment if for
any reason you need to move away from 4D for your data storage.
6. Develop new screens to display data from your new Business Objects.
Test with both the old data structure where appropriate and the new
structure to demonstrate that you haven't broken anything.

Of course, if you are developing new features, these will only apply
to the new data design.


HTH

David Dancy
Sydney, Australia
**********************************************************************
4D Server v11 SQL has arrived!
Buy it NOW at http://store.4ddepot.com

4th Dimension Internet Users Group (4D iNUG)
FAQ: http://www.4d.com/support/faqnug.html
Archive: http://dir.gmane.org/gmane.comp.lang.inug-4d.tech
Unsub: mailto:4D_Tech-Unsubscribe-d2/MUvgItPNWk0Htik3J/***@public.gmane.org
Post: mailto:4d_tech-jTdV3SBRT1BWk0Htik3J/***@public.gmane.org
Options: https://lists.4d.com/mailman/listinfo/4d_tech
**********************************************************************
Mark Jeffreys
2008-09-09 00:56:33 UTC
Permalink
<snip>

David:

Thanks for your suggestions!
Post by David Dancy
1. Don't throw out the old stuff. It's working!
2. Develop the new stuff in v11 in a converted database that has all
the old stuff (including tables) still in it...
3. Develop the new stuff just-how-you-like-it, with your ideal
structures, normalised tables, well-named key fields, and so on,
_alongside the old stuff_. Then write data conversion methods to
convert the old data to your new design.
OK, I can see the wisdom in this approach; it¹s more or less what I did in
the last major rewrite. I kind of like the idea of starting off with a blank
slate, and no worries about conversion or compatibility modes. But with the
improvements in Design mode in recent years, I suppose it shouldn¹t be as
hard to keep the new stuff and the old stuff separate.
Post by David Dancy
4. Abstract a Data Access Layer out of your existing code so that it
deals with Business Objects rather than records...
I think I understand the concept here, but I¹m lacking any specifics. What
I¹ve read about layered architecture makes a lot of sense, but since all the
references are for languages in which I¹m not fluent and for databases that
don¹t have anything like the integrated environment of 4D, I don¹t know
where to start. Can you give me a simple (even simplistic) example?
Post by David Dancy
...Then write code to fill in the Business Objects from both
your old and new data structures and save them back. This will also
provide your customers with a transparent upgrade path, and it won't
require a massive export-import operation for them. Your new program
can read data from the old design and write it back to the new design
without the customer being any the wiser, especially if you make the
old tables invisible.
I see how this follows from the previous two sections, but I¹m wondering if
this is overkill. Remember, I¹m an in-house developer and this is a single
database. Obviously, I have to keep my boss and my users happy, but nobody
except me cares about the actual process of conversion. I can see that this
approach might be beneficial in verifying that the new and old structures
are functionally the same, but at first glance it would seem to mean having
to write twice as much code in whatever layer does the translation between
the processing code and the two different structures. I¹m guessing that you
think there are benefits to this that I¹m not seeing yet.
Post by David Dancy
5. Use SQL where you can for data access, unless the 4D commands are
genuinely superior or you need the convenience of a Current Selection...
I¹m rather looking forward to playing with SQL again; it¹s been over 15
years since I used it with any regularity, and I¹ve forgotten most of what I
once knew. Any performance gains will be icing on the cake. On the other
hand, having a Current Selection has become so ingrained I¹m probably going
to find myself assuming I have one whether it's there or not.
Post by David Dancy
6. Develop new screens to display data from your new Business Objects.
Test with both the old data structure where appropriate and the new
structure to demonstrate that you haven't broken anything.
That certainly makes sense, given the architecture discussed above. If I get
that far, at least I don¹t think I¹d have any problems with this! :)

David, thanks again for your thoughtful reply,

-- MLJ
--
Mark L. Jeffreys
Senior Staff Technology Solutions
Creative Services
Charles Schwab & Co., Inc.
San Francisco CA

Warning: All e-mail sent to this address will be received by the Charles
Schwab & Co. corporate e-mail system and is subject to archival and review
by someone other than the recipient.

**********************************************************************
4D Server v11 SQL has arrived!
Buy it NOW at http://store.4ddepot.com

4th Dimension Internet Users Group (4D iNUG)
FAQ: http://www.4d.com/support/faqnug.html
Archive: http://dir.gmane.org/gmane.comp.lang.inug-4d.tech
Unsub: mailto:4D_Tech-Unsubscribe-d2/MUvgItPNWk0Htik3J/***@public.gmane.org
Post: mailto:4d_tech-jTdV3SBRT1BWk0Htik3J/***@public.gmane.org
Options: https://lists.4d.com/mailman/listinfo/4d_tech
**********************************************************************
David Dancy
2008-09-09 03:45:00 UTC
Permalink
Sorry Mark, I'd forgotten that yours is an internal application. Our
situation is a little different, where we have nearly 2000
installations around Oz to think about. :) But you may still find the
following worth evaluating for your situation.

The rationale behind having two implementations of the data access
when you're doing a new design is that you can then build the new
design piecemeal, and you don't have to have an "all or nothing" Big
Switchover. Each portion of the database can be abstracted into data
access methods for the old design. You can then design the new
structure and create new data access methods to take that abstraction
and put it back into the new structure, and your switchover to the new
system will be totally seamless. The abstraction then serves as the
interface between the two models of the world. It's not twice as much
code (at least, I don't think so). It's more like this:

OldTableX OldTableY OldTableZ
| | |
-----------------------------------------
|
OldAccessMethods
|
Abstraction
|
NewAccessMethods
|
--------------------------------------------------------------
| | | |
NewTableA NewTableB NewTableC NewTableD

I agree that the abstraction could be overkill in anyone's situation,
for sure. However, it does create what to me is an important advantage
in your database, where instead of thinking about records individually
you can think about a "concept" that makes sense to the end user. As
the developer you then work with those concepts and provide
translations from the concept to the underlying data. In turn this
means that the underlying data structure can change (as you are
proposing in this case) or even move between database engines and the
user just sees the same-old same-old happening as per usual. It's a
little harder to do in 4D because we don't have Objects (sorry, that
old chestnut again) but you can approximate something like them by
using other techniques. At the very least I would create a series of
methods whose job it is to load and save stuff for each form as a
whole, because the form is what the user deals with. Then when the
form changes, you change the Load/Save methods and it should all still
work properly.

To frame a more concrete example, suppose you have an Invoice data
structure in your old design where some data is denormalised, and you
want a new design where it is properly normalised. Your abstraction is
then a representation of the concept of an Invoice, which includes at
least everything users see when they are interacting with the Invoice
form. You then write a method LoadOldInvoice(InvoiceID) which
populates the form from the old data, and a method SaveOldInvoice
which puts it back. You re-design your data structure to normalise the
data, and then you write two more methods LoadNewInvoice(InvoiceID)
and SaveNewInvoice to load and save the same abstraction back to the
new data structure. If you then implement the form with variables
(instead of fields) and arrays in Listboxes or AreaLists, you can use
SQL to load the data and save it back, and it matters not what
database engine you have used to store it. The "abstraction" in this
case is the series of variables and arrays that you need to describe
the Invoice Business Object, most of which will be visible and
enterable on the form, but not all. The collection of all these
variables and arrays together would be part of a (Business) Object in
other languages; in 4D, you have to make your own barriers between
things like this by using a strict module-naming convention. This in
turn has the advantage that you don't need to have actual records
present to be able to test the access methods; you can have a separate
test data file on (e.g.) SQL Server Express and talk to that instead.
Plus, you get data conversion "for free" because it is just
LoadOldInvoice(X) then SaveNewInvoice() for each InvoiceID X in your
database.

If you want to use fields on your forms instead of variables (for
example, because they have a built-in length restriction or you're
concerned about the potential size of your process variable list), you
will need to create extra forms to display the new data structure, but
the principle will be much the same. This is certainly simpler to
implement, but a bit less flexible. It may be better in your
particular case to just do this and then provide a data conversion
between the old and new data structures. Then after converting you
flick a switch to say "use the new form" and there's nothing more to
do.

However, to me, there are some interesting advantages of the full-on
Business Objects abstraction I've tried to illustrate:
1. You get to think in the same terms that your users do. This is a
good thing for the design of the system.
2. Your database gets to work with tables that suit it. If a design
proves problematic for any reason, you can change it (obviously also
updating the data access methods) without affecting the forms that the
user interacts with. In v11, you won't run out of tables, so you can
make changes quite freely as your needs evolve.
3. The data can be stored anywhere: 4D, 4D Server, XML, flat files,
SQLLite, SQL Server, loaded live from Web Services off the Internet,
etc. The form neither knows nor cares. It only deals with the
variables and arrays that it displays. It hands off responsibility for
the permanent storage of the data to a method that knows where things
belong and how to put them there and get them back.
4. You can test the form's variables, form method, data access
methods, and so on without needing to load up the form and push
buttons by hand (you can do this if you use fields too).
5. You can design your own data structure to manage a Business Object
while it's in between the form and the database. You could use XML,
ObjectTools, or just make up a naming convention. XML or OT would be
nice, because you could then also "clone" a Business Object very
easily. You could then give your users a "Save As..." facility where
they could take (e.g.) one Job as a "template" for another one. It
would be hardly any work to implement that with a Business Object, but
quite tricky if you're manipulating 4D records directly.

Using SQL to load data is really straightforward, but using SQL to
save it back to the database is not, as I'm sure you're aware. SQL
doesn't have the concept of a Current Record, so it won't lock records
in the same way as 4D does. You have to lock them explicitly, or have
some other mechanism to ensure that you are managing concurrency
correctly. But this is a wheel that has been invented many times
already, and you can just pick whichever solution you want.

You alone will know if any of these ideas is going to be too much work
to be worthwhile. I will be interested in your conclusions!

HTH

David
**********************************************************************
4D Server v11 SQL has arrived!
Buy it NOW at http://store.4ddepot.com

4th Dimension Internet Users Group (4D iNUG)
FAQ: http://www.4d.com/support/faqnug.html
Archive: http://dir.gmane.org/gmane.comp.lang.inug-4d.tech
Unsub: mailto:4D_Tech-Unsubscribe-d2/MUvgItPNWk0Htik3J/***@public.gmane.org
Post: mailto:4d_tech-jTdV3SBRT1BWk0Htik3J/***@public.gmane.org
Options: https://lists.4d.com/mailman/listinfo/4d_tech
**********************************************************************
Garri Ogata
2008-09-09 03:14:21 UTC
Permalink
Mark,
If you have the "real luxury" of starting from scratch. I would start from scratch. If it started from Filemaker and was changed and modified along the way then its more than likely the structure isn't very good to begin with. How could it be? You are talking about something that has started as a flat file application. I'm sure what you have done is nice. But why continue down the same path you started 16 years ago.

The real question is how much time do you have to do this "major rewrite" or "start from scratch"? Based on that you should decide what is the best thing to do. If your lucky enough that you can plan out a real rewrite. Plan it out and do it. Why waste good programming time on a poor structure?

Garri
Date: Mon, 8 Sep 2008 16:09:42 -0700
Subject: Starting from Scratch - What Would You Do?
The recent thread "Things I've learned [Long]" got me thinking...
Sometime in the next few months, it looks like I'll have the chance to do
something I've been wanting to do for a long time, that being a major
rewrite of JobTrack, our main production database in Creative Services (i.
e., the in-house advertising & interactive agency). I started working on
this in 1992, when it was still quite small; someone in the department had
begun writing it in FileMaker but then realized that FileMaker wasn't up to
the task and started rewriting it in 4D. I was called in to assist and have
been working on it ever since, first as a part-time contractor, then
full-time, and in recent years as an employee. It was initially based on a
skeleton I had developed in version 3 when I was working at UCSF, which was
itself based on a version 2 database, which was in turn based on the first
DB I wrote in version 1 back in 1987-88. (These are all the US version
numbers, by the way.)
JobTrack has grown from a relatively simple tool that only recorded general
information about jobs to a mission-critical app that is used to track staff
hours, staff scheduling, interaction with other tools and interfaces on the
intranet, various accounting and billing functions and reports, management
overviews and on and on. Although I did a fairly extensive rewrite back in
1999-2000 (when we went to version 6, IIRC), there is still quite a lot of
code that dates back to the version 3 days, and maybe even some left from
version 2 here and there. What I wasn't able to do in that rewrite was to
make some of the larger structural changes that I would have liked to make
-- change some things that even back then I knew were undesirable, and now
know to be hampering my ability to maintain and extend JobTrack. Many of
these were due to either design decisions made early on, some even before I
came on board, or were due to the fact that my skill and experience in
design and programming just were not as developed back then. (As a random
but important example, using "meaningful" table keys -- man, that's caused
me a lot of grief!). Another consideration is getting rid of a lot of
unnecessary bloat, such as various denormalizations and summary tables that
were required for decent performance 10 years ago but now just complicate
matters.
So I now hope to have an opportunity not many of us get: to start
more-or-less from scratch, and rebuild JobTrack in a manner that will allow
me to spend less time tiptoeing through the shaky foundations and more time
actively extending and improving it. Now that I'm starting to delve
seriously into v11, with all the new options available therein, it seems
like the right time to start fresh. I've been thinking about how to approach
this, given all the paradigms and techniques I've encountered over the years
from the iNUG and other sources. I'd like to start off with a list of
principles -- what I should and shouldn't do -- and I'd like your advice as
- Unit testing/automated testing
- Strict modularization
- Layered architecture (3-tier? MVC? Other?)
- Function wrapping
- Accessor methods for global(ish) values
- Ubiquitous error handling
- Source control
- Custom data structures where appropriate
- Administrative interface *
I've been using some of these principles for years as I've expanded or
updated parts of the code base (modules, error handling, accessors). Others
I'm aware of as concepts but would like to have specific examples and advice
for implementation, especially unit/automated testing and layered
architecture. And I'm sure there are many important items I've left off this
initial list.
Note that I'm not trying to provoke arguments about the superiority of one
approach over another; I'm trying to compile a list of things I shouldn't
ignore without due consideration. I don't want to get some months into a
huge undertaking only to realize that I've designed or coded myself into a
corner because I missed something I should have done at the beginning.
That's it -- where would you start and what would you do if you had a chance
to do it all over again?
Thanks in advance,
-- MLJ
* I've always used the User environment for a lot of ad hoc reporting and
data massaging on the live (compiled) server, and it's gone away in v11.
Although that wouldn't matter nearly as much if I could set the standard
Query Editor in Application mode to remember what had been entered when last
used!
--
Mark L. Jeffreys
Senior Staff Technology Solutions
Creative Services
Charles Schwab & Co., Inc.
San Francisco CA
Warning: All e-mail sent to this address will be received by the Charles
Schwab & Co. corporate e-mail system and is subject to archival and review
by someone other than the recipient.
**********************************************************************
4D Server v11 SQL has arrived!
Buy it NOW at http://store.4ddepot.com
4th Dimension Internet Users Group (4D iNUG)
FAQ: http://www.4d.com/support/faqnug.html
Archive: http://dir.gmane.org/gmane.comp.lang.inug-4d.tech
Options: https://lists.4d.com/mailman/listinfo/4d_tech
**********************************************************************
_________________________________________________________________
See how Windows Mobile brings your life together—at home, work, or on the go.
http://clk.atdmt.com/MRT/go/msnnkwxp1020093182mrt/direct/01/**********************************************************************
4D Server v11 SQL has arrived!
Buy it NOW at http://store.4ddepot.com

4th Dimension Internet Users Group (4D iNUG)
FAQ: http://www.4d.com/support/faqnug.html
Archive: http://dir.gmane.org/gmane.comp.lang.inug-4d.tech
Unsub: mailto:4D_Tech-Unsubscribe-d2/MUvgItPNWk0Htik3J/***@public.gmane.org
Post: mailto:4d_tech-jTdV3SBRT1BWk0Htik3J/***@public.gmane.org
Options: https://lists.4d.com/mailman/listinfo/4d_tech
**********************************************************************
Randy Jaynes
2008-09-09 03:42:34 UTC
Permalink
Hi, Mark!

If I were in your shoes, I would take a look at StructurePulse to help
you plan this out. I think David's suggestions are right on the money,
and StructurePulse could help you along the way.

It can do a lot to help illuminate things for you.

Randy

---------------------------------
Randy Jaynes
Senior Programmer

PrintPoint, Inc.
57 Ludlow Lane
Palisades, NY 10964
USA

Tel: 845 359-0298
Fax: 845 359-3468
Post by Garri Ogata
Mark,
If you have the "real luxury" of starting from scratch. I would
start from scratch. If it started from Filemaker and was changed
and modified along the way then its more than likely the structure
isn't very good to begin with. How could it be? You are talking
about something that has started as a flat file application. I'm
sure what you have done is nice. But why continue down the same
path you started 16 years ago.
The real question is how much time do you have to do this "major
rewrite" or "start from scratch"? Based on that you should decide
what is the best thing to do. If your lucky enough that you can
plan out a real rewrite. Plan it out and do it. Why waste good
programming time on a poor structure?
Garri
Date: Mon, 8 Sep 2008 16:09:42 -0700
Subject: Starting from Scratch - What Would You Do?
The recent thread "Things I've learned [Long]" got me thinking...
Sometime in the next few months, it looks like I'll have the chance to do
something I've been wanting to do for a long time, that being a major
rewrite of JobTrack, our main production database in Creative
Services (i.
e., the in-house advertising & interactive agency). I started
working on
this in 1992, when it was still quite small; someone in the
department had
begun writing it in FileMaker but then realized that FileMaker wasn't up to
the task and started rewriting it in 4D. I was called in to assist and have
been working on it ever since, first as a part-time contractor, then
full-time, and in recent years as an employee. It was initially based on a
skeleton I had developed in version 3 when I was working at UCSF, which was
itself based on a version 2 database, which was in turn based on the first
DB I wrote in version 1 back in 1987-88. (These are all the US version
numbers, by the way.)
JobTrack has grown from a relatively simple tool that only recorded general
information about jobs to a mission-critical app that is used to track staff
hours, staff scheduling, interaction with other tools and
interfaces on the
intranet, various accounting and billing functions and reports, management
overviews and on and on. Although I did a fairly extensive rewrite back in
1999-2000 (when we went to version 6, IIRC), there is still quite a lot of
code that dates back to the version 3 days, and maybe even some left from
version 2 here and there. What I wasn't able to do in that rewrite was to
make some of the larger structural changes that I would have liked to make
-- change some things that even back then I knew were undesirable, and now
know to be hampering my ability to maintain and extend JobTrack. Many of
these were due to either design decisions made early on, some even before I
came on board, or were due to the fact that my skill and experience in
design and programming just were not as developed back then. (As a random
but important example, using "meaningful" table keys -- man, that's caused
me a lot of grief!). Another consideration is getting rid of a lot of
unnecessary bloat, such as various denormalizations and summary tables that
were required for decent performance 10 years ago but now just complicate
matters.
So I now hope to have an opportunity not many of us get: to start
more-or-less from scratch, and rebuild JobTrack in a manner that will allow
me to spend less time tiptoeing through the shaky foundations and more time
actively extending and improving it. Now that I'm starting to delve
seriously into v11, with all the new options available therein, it seems
like the right time to start fresh. I've been thinking about how to approach
this, given all the paradigms and techniques I've encountered over the years
from the iNUG and other sources. I'd like to start off with a list of
principles -- what I should and shouldn't do -- and I'd like your advice as
- Unit testing/automated testing
- Strict modularization
- Layered architecture (3-tier? MVC? Other?)
- Function wrapping
- Accessor methods for global(ish) values
- Ubiquitous error handling
- Source control
- Custom data structures where appropriate
- Administrative interface *
I've been using some of these principles for years as I've expanded or
updated parts of the code base (modules, error handling,
accessors). Others
I'm aware of as concepts but would like to have specific examples and advice
for implementation, especially unit/automated testing and layered
architecture. And I'm sure there are many important items I've left off this
initial list.
Note that I'm not trying to provoke arguments about the superiority of one
approach over another; I'm trying to compile a list of things I shouldn't
ignore without due consideration. I don't want to get some months into a
huge undertaking only to realize that I've designed or coded myself into a
corner because I missed something I should have done at the
beginning.
That's it -- where would you start and what would you do if you had a chance
to do it all over again?
Thanks in advance,
-- MLJ
* I've always used the User environment for a lot of ad hoc
reporting and
data massaging on the live (compiled) server, and it's gone away in v11.
Although that wouldn't matter nearly as much if I could set the standard
Query Editor in Application mode to remember what had been entered when last
used!
--
Mark L. Jeffreys
Senior Staff Technology Solutions
Creative Services
Charles Schwab & Co., Inc.
San Francisco CA
Warning: All e-mail sent to this address will be received by the Charles
Schwab & Co. corporate e-mail system and is subject to archival and review
by someone other than the recipient.
**********************************************************************
4D Server v11 SQL has arrived!
Buy it NOW at http://store.4ddepot.com
4th Dimension Internet Users Group (4D iNUG)
FAQ: http://www.4d.com/support/faqnug.html
Archive: http://dir.gmane.org/gmane.comp.lang.inug-4d.tech
Options: https://lists.4d.com/mailman/listinfo/4d_tech
**********************************************************************
_________________________________________________________________
See how Windows Mobile brings your life together—at home, work, or on the go.
http://clk.atdmt.com/MRT/go/msnnkwxp1020093182mrt/direct/01/**********************************************************************
4D Server v11 SQL has arrived!
Buy it NOW at http://store.4ddepot.com
4th Dimension Internet Users Group (4D iNUG)
FAQ: http://www.4d.com/support/faqnug.html
Archive: http://dir.gmane.org/gmane.comp.lang.inug-4d.tech
Options: https://lists.4d.com/mailman/listinfo/4d_tech
**********************************************************************
**********************************************************************
4D Server v11 SQL has arrived!
Buy it NOW at http://store.4ddepot.com

4th Dimension Internet Users Group (4D iNUG)
FAQ: http://www.4d.com/support/faqnug.html
Archive: http://dir.gmane.org/gmane.comp.lang.inug-4d.tech
Unsub: mailto:4D_Tech-Unsubscribe-d2/MUvgItPNWk0Htik3J/***@public.gmane.org
Post: mailto:4d_tech-jTdV3SBRT1BWk0Htik3J/***@public.gmane.org
Options: https://lists.4d.com/mailman/listinfo/4d_tech
**********************************************************************
Mark Jeffreys
2008-09-09 18:34:12 UTC
Permalink
Post by Garri Ogata
If you have the "real luxury" of starting from scratch. I would start from
scratch. If it started from Filemaker and was changed and modified along the
way then its more than likely the structure isn't very good to begin with.
How could it be? You are talking about something that has started as a flat
file application. I'm sure what you have done is nice. But why continue down
the same path you started 16 years ago.
The real question is how much time do you have to do this "major rewrite" or
"start from scratch"? Based on that you should decide what is the best thing
to do. If your lucky enough that you can plan out a real rewrite. Plan it
out and do it. Why waste good programming time on a poor structure?
Garri:

I agree completely! I think half of the problems I have are directly related
to the structure, which grew amorphously over the years, sometimes in
conflicting directions. Especially in the years before I worked here full
time, the emphasis was on getting it working correctly as quickly as
possible, with little time for future considerations. The first thing I plan
to do is to design a rational structure. It may not be perfect, and it will
probably have to be modified as I go along, but it will certainly be better
than what I have now. And I¹m hoping to have all the time I need since (so
far) I don¹t have to work to a deadline for this; then again, when do we
ever end up with *all* the time we need? :)

Thanks,

-- MLJ
--
Mark L. Jeffreys
Senior Staff Technology Solutions
Creative Services
Charles Schwab & Co., Inc.
San Francisco CA

Warning: All e-mail sent to this address will be received by the Charles
Schwab & Co. corporate e-mail system and is subject to archival and review
by someone other than the recipient.

**********************************************************************
4D Server v11 SQL has arrived!
Buy it NOW at http://store.4ddepot.com

4th Dimension Internet Users Group (4D iNUG)
FAQ: http://www.4d.com/support/faqnug.html
Archive: http://dir.gmane.org/gmane.comp.lang.inug-4d.tech
Unsub: mailto:4D_Tech-Unsubscribe-d2/MUvgItPNWk0Htik3J/***@public.gmane.org
Post: mailto:4d_tech-jTdV3SBRT1BWk0Htik3J/***@public.gmane.org
Options: https://lists.4d.com/mailman/listinfo/4d_tech
**********************************************************************
Mark Jeffreys
2008-09-09 18:39:25 UTC
Permalink
... The rationale behind having two implementations of the data access
when you're doing a new design is that you can then build the new
design piecemeal, and you don't have to have an "all or nothing" Big
Switchover...
<big snip>
David:

Thanks for a concise and cogent explanation of a complex concept. I now have
a much better idea of how to implement your take on abstraction. I also see
how the ideas about using custom data structures and function wrapping that
I've been absorbing over the years (thanks esp. to David A. & Jack des B.)
would fit into this.

I suspect it will take a while and a bit of experimenting to get it working
-- and more importantly, to get comfortable with a different way of looking
at the flow of control in the code. This post has enough to keep me busy for
a month at least! (Though I'm sure I'll have more questions before long.)

Thanks again,

-- MLJ
--
Mark L. Jeffreys
Senior Staff Technology Solutions
Creative Services
Charles Schwab & Co., Inc.
San Francisco CA

Warning: All e-mail sent to this address will be received by the Charles
Schwab & Co. corporate e-mail system and is subject to archival and review
by someone other than the recipient.

**********************************************************************
4D Server v11 SQL has arrived!
Buy it NOW at http://store.4ddepot.com

4th Dimension Internet Users Group (4D iNUG)
FAQ: http://www.4d.com/support/faqnug.html
Archive: http://dir.gmane.org/gmane.comp.lang.inug-4d.tech
Unsub: mailto:4D_Tech-Unsubscribe-d2/MUvgItPNWk0Htik3J/***@public.gmane.org
Post: mailto:4d_tech-jTdV3SBRT1BWk0Htik3J/***@public.gmane.org
Options: https://lists.4d.com/mailman/listinfo/4d_tech
**********************************************************************
Mark Jeffreys
2008-09-10 01:01:31 UTC
Permalink
Post by Randy Jaynes
Hi, Mark!
If I were in your shoes, I would take a look at StructurePulse to help
you plan this out. I think David's suggestions are right on the money,
and StructurePulse could help you along the way.
It can do a lot to help illuminate things for you.
Randy
Hey, Randy -- it's been a while!

I have licenses for DataPulse & StructurePulse, but I had forgotten all
about them! I haven¹t used them in any meaningful way since I removed all
the subtables from my database, lo these many years ago. Thanks for the
reminder; I'll check them out.

-- MLJ
--
Mark L. Jeffreys
Senior Staff Technology Solutions
Creative Services
Charles Schwab & Co., Inc.
San Francisco CA

Warning: All e-mail sent to this address will be received by the Charles
Schwab & Co. corporate e-mail system and is subject to archival and review
by someone other than the recipient.

**********************************************************************
4D Server v11 SQL has arrived!
Buy it NOW at http://store.4ddepot.com

4th Dimension Internet Users Group (4D iNUG)
FAQ: http://www.4d.com/support/faqnug.html
Archive: http://dir.gmane.org/gmane.comp.lang.inug-4d.tech
Unsub: mailto:4D_Tech-Unsubscribe-d2/MUvgItPNWk0Htik3J/***@public.gmane.org
Post: mailto:4d_tech-jTdV3SBRT1BWk0Htik3J/***@public.gmane.org
Options: https://lists.4d.com/mailman/listinfo/4d_tech
**********************************************************************
John Foster
2008-09-10 15:26:21 UTC
Permalink
Hey Mark,

I haven't followed this thread closely enough so I am not sure of
what the issues were. If you want to discuss anything related to
DataPulse or StructurePulse please feel free to contact me directly.

John...
Post by Mark Jeffreys
Post by Randy Jaynes
Hi, Mark!
If I were in your shoes, I would take a look at StructurePulse to
help
you plan this out. I think David's suggestions are right on the
money,
and StructurePulse could help you along the way.
It can do a lot to help illuminate things for you.
Randy
Hey, Randy -- it's been a while!
I have licenses for DataPulse & StructurePulse, but I had forgotten
all
about them! I haven¹t used them in any meaningful way since I
removed all
the subtables from my database, lo these many years ago. Thanks for
the
reminder; I'll check them out.
--------------------------------------------------------------

John Foster
Eternity Software
425-486-1622

http://www.eternity-software.com/ES_Products.html

--------------------------------------------------------------


**********************************************************************
4D Server v11 SQL has arrived!
Buy it NOW at http://store.4ddepot.com

4th Dimension Internet Users Group (4D iNUG)
FAQ: http://www.4d.com/support/faqnug.html
Archive: http://dir.gmane.org/gmane.comp.lang.inug-4d.tech
Unsub: mailto:4D_Tech-Unsubscribe-d2/MUvgItPNWk0Htik3J/***@public.gmane.org
Post: mailto:4d_tech-jTdV3SBRT1BWk0Htik3J/***@public.gmane.org
Options: https://lists.4d.com/mailman/listinfo/4d_tech
**********************************************************************
Continue reading on narkive:
Loading...