Discussion:
D3.js Wow. Some notes on SVG, D3, and data visualizations
(too old to reply)
David Adams
2015-12-18 00:12:10 UTC
Permalink
Almost a month back I wrote in and asked for some suggestions related to
data visualizations, SVG, and JavaScript. Starting from, well, nowhere I
was very kindly given a stack of excellent suggestions. I've been digging
in and thought I'd pass along a few thoughts and links for others and the
archives. This isn't super well organized and other people will know a lot
more than me about every part of this post...hopefully someone will think
to add some bits and pieces. If I've gotten something wrong, feel free to
offer corrections.

For background, every couple of years I seem to run into something gets me
really excited and fanatical. To those I've abused in the past with my
fanaticism, apologies. To those of you I'm about to abuse...apologies...but
you've been warned. Here's a quick summary: SVG isn't hard, D3 is
mind-bendingly fantastic and, as it turns out, easy to get a lot done with
in a hurry. How does all of this fit in with 4D? That depends on what
you're doing. In my case, I'm using 4D to store, grind, and search a lot of
data and then spit it out in data formats that can be charted.

-------
SVG
-------
First thing's first: SVG. I noticed an email the other day that Keisuke
Miyako is presenting a component of the Portland Summit post training
called "SVG Cookbook For 4D Programmers." I can't find a direct link to the
write-up, but this gets you to the top page, then click "Post-class" and
then click on "Read More" next to the module description:

http://summit.4d.com/us_en/training/

Chances are, this part of the course will be worth the price of admission
on its own.

In my case, I'd never messed around with SVG so I thought that I'd try it
out by hand using a text editor and then building up to some code that
spits out text. I still haven't looked at 4D's SVG commands and,
ultimately, am more likely to use 4D to send out data that a JavaScript
will process into SVG, as needed. I *highly recommend* trying out some SVG
production by hand. Make some overlapping squares or something - try
building a tiny bar chart. Why? Well, you'll see in a big hurry that basic
SVG is easy to read and write and is pretty darn useful. You'll also
probably quickly see that writing tons of SVG by hand is tedious and that
it begs for automation. Some tips:

* With SVG, you declare what you want and a renderer in the browser
interprets your instructions and generated a graphic. Nice.

* SVG is clickable. Actually, that's just the headline - SVG nodes have a
DOM and actions (hover, mouseover, click, etc.) can be attached to nodes.
You can bake the assignments right into the raw SVG or you can assign them
with JavaScript. Either kind of manually/tediously or by using a jQuery/D3
style selection. (D3 has a really sweet approach to selections and method
chaining that make selecting a bunch of objects one line of code and then
iterating over them one more line of code. Making the iteration loop
invisible is pretty nice.)

* Oh yeah, the DOM. With JavaScript you always here about "the DOM." For
the uninitiated, that means the Document Object Model...which is a
meaningless phrase, IMO. It's a tree of nodes in the document that you can
read, write, and modify through JavaScript. An external SVG file has its
own DOM with some slight and subtle difference from the standard HTML DOM.
Most CSS selectors are the same...but not all. These are all the sorts of
details I'll want to avoid...so...if you go down this road, I suggest:

-- Consider using the HTML5 <svg> tag as then the SVG is part of the parent
page's DOM without any fuss.

-- Yeah, single-origin policy restrictions are a pain. For much of this
stuff, you pretty well need to have a Web server running even if you're
working locally. 4D can be a great help here! It's got a nice Web server
built in. You can also use Apache or whatever.

-- Get a library to take care of the tedium....D3....

* If you do use stand-alone SVG, know that you can include CSS and
JavaScript directly in the SVG file itself. This makes the JavaScript and
the SVG nodes all part of the same node tree (DOM) so there's no hassle
there.

* There are a lot of great SVG resources on-line but the one I keep going
back to is here:

http://tutorials.jenkov.com/svg/index.html

Microsoft's pages are also good but don't have that personal touch.

* Lots of SVG code includes an element named <g> for group. It's not a
great name, but it's short...the same (or a similar) concept is found in
any graphics programming environment but with a different name. The idea is
that you get to group elements so that you can operate on them as a block.
Say you've got a square and some text - you can rotate them as a group.
Beyond that, <g> lets you work with a local coordinate space and then
offset it in a block. Eh? Say you're drawing horizontal bars. You can keep
everything oriented to a local y without worrying if you're the first bar
or the fiftieth. Within the object, you can then move the entire block
around to position it vertically. Here's what it looks like to move a block
down 100 units (you can set your own and even mix units, depending on your
needs):

<g transform="translate(0,100)">

At first I didn't appreciate <g> and ignored it. It turns out that when
you're looping over a data source to generate objects, it's pretty darn
handy not to have to keep track of an overall block of object's position in
the final composition. This will probably only make sense if you're trying
to code this stuff, but I toss it out there so that it's in the back of
people's minds. Oh, and going this route can also help avoid some hassles
with alignment that are easy to get when laying out text against other
objects like lines or rectangles.

Tip: Big graph? Put it into a smaller <div> that's set to show scrollbars
when the contents are too big to display completely. Works fine.

-------
D3
-------
D3 (http://d3js.org/) which stands for "Data Driven Documents", is a very
big noise in the data visualization world. To connect things up, it's
generating SVG underneath it all and you can get at the generated SVG
nodes/objects, if you need to. So, it's kind of good to have an idea of how
SVG works underneath. Not necessarily required, but helpful. D3 has had
some big upgrades in the last few years so it's worth another look if you
considered and rejected it years ago.

I also had a look at Raphael (http://raphaeljs.com/) and HighCharts (
http://www.highcharts.com/), both of which look good. Actually, HighCharts
looks awesome but I figured I'd go with the open-source D3...but this is a
hobby project...for a commercial project, it would be worth looking at
HighCharts. Many of you will remember that Stephen Orth pretty much got a
"best in show" award at the 2013 Summit for his High Charts presentation.
For attendees, you can revisit the presentation at:

http://kb.4d.com/assetid=77104

Anyway, I didn't go with Raphael as it may be a dead end now. At the least,
it has nothing close to the momentum of D3. Nothing does, so far as I can
tell. A bunch of thoughts:

* Amazing piece of work. Just amazing.

* There are thousands of examples and a very active community. Here's a
starting point that's not as overwhelming as some:
https://github.com/mbostock/d3/wiki/Gallery

* Try this tutorial:

http://bost.ocks.org/mike/bar/

You'll want a very minimal level of JavaScript knowledge (very basic), a
text editor, and a browser.

* It's easy to install. Not a big deal to some, but dealing with JS library
dependencies is sometimes an enormous pain so I appreciate something that's
easy to install. You copy in a library and link to it from your script. And
now you're done.

* D3 is a platform and tons of libraries have been built on top. I've been
experimenting with dimple (http://dimplejs.org/) which makes it a snap to
produce some complex charts with a few lines of code. As an example, I've
been using 4D to download and grind up data sets related to global bird
distributions at various taxonomic levels. I've got hundreds of thousands
of sums and comparisons stored that I can export as graphable series. One
complex bubble chart graphs this data:

-- The y axis is the overall land area of the location. So, Brazil is way
up top and Belgium is pretty far down.
-- The x axis shows degrees of similarity with a starting point. So, Canada
is very similar to the lower 48 and not very similar to Australia.
-- The bubbles are sized based on total species counts, so Peru gets a huge
bubble and New Zealand gets a small bubble.
-- ...and then each bubble is colored based on ecological region.

How much JavaScript does that all take? About ten lines...and some of those
are decorative. In this case, "decorative" means formatting numbers and
setting axis labels...no "chart junk." So, a few lines of code to get what
Ed Tufte called "high data density."

* Speaking of Saint Tufte, I know that many of the old guard have attended
his trainings or read his books. Well, the D3 guys certainly have. They're
part of the modern information visualization movement that's about
*respecting the data.* I think that the main guy is now at The New York
Times. There are no pointless drop shadows, no three dimensional graphs of
two dimensions of data (Hello! Excel, I'm looking at you) and a real
emphasis on getting the math right for proportionate displays. So, yeah,
this is the real deal. Once you've got the hang of it a bit, it's far
simpler (and better) than charting in, well, Excel.

* A quick word about data. If you're using D3, you can pull in data using a
variety of formats including text, HTML page elements, JSON (or "JSON with
padding"), xml, csv, or tsv (tab-separated values). Some comments:

-- JSON is super nice because then each element is typed correctly.

-- JSON is terrible because the "name": part of each element is repeated
over and over. And over. If you have a big data set, perhaps TSV is a good
plan.

-- Single-origin policy! Grrr. Browser complain. Well, they complain at the
console, in the standard Web window you just don't get what you expect.

-- TSV (etc.) is great because it's so compact.

-- TSV (etc.) is great because you can have numbers with commas and so on.

-- TSV (etc.) is a pain because everything is imported as a string. There's
no schema or meta-data to guide D3 in how to interpret the data. You can do
a post conversion within your JS, like this fragment:

d3.tsv('./bubble_data/locations.tsv', function(data) {
/* Convert strings to numbers. If you use JSON instead of TSV, you don't
need this step. */
data.forEach(function(d){
d['Total Species'] = +d['Total Species'];
d['Shared Species'] = +d['Shared Species'];
d['Shared Percent'] = +d['Shared Percent'];
d['Endemic Percent'] = +d['Endemic Percent'];
d['Square KMs'] = +d['Square KMs'];
})

Is that a good example? I don't know...it might be a terrible example, but
it does show the kind of code to look for. Note that there's a solid
argument saying that this make the JS and the source data a bit to
intimate. Nice to have clean data with type definitions inside of it that
is simply passed through to D3. If you feel this way, use JSON.

-- D3 has a filter() function that lets you extract series from your import
source. So, you can have a shared data set up on the server and then pull
out only the data range or columns you need, for example. (The D3 examples
filter by date but that's not the only option...it's just a good example.)

-------
Tools
-------
Right, you need tools to work with JavaScript. In my case, what bugs me
most is not having a great integrated syntax checker like 4D's. It. Makes.
Me. Uncomfortable. I've tried a bunch of tools and have a bunch more to
try. For now, what's working okay is Microsoft's Visual Studio Code +
ESLint. ESLint is the secret sauce. It's an easy-to-install extension that
highlights errors and warnings. What errors and warnings? That's up to you.
It will flag all sorts of tiny formatting problems (no space after function
name!) if you want it to, but it's deeply configurable. It's worth sitting
down with a few small scripts and messing with the config for 45 minutes
until you've got the level of errors you like. It's still early days for
Visual Studio Code on OS X but it's stable. But no cold folding yet :(

I also like BBEdit but, well, it's just not all that when it comes to some
of this.

Every WebKit based browser on earth has nice developer tools built in now.
Nice. Visual Studio Code has a debugger that installs in Chrome but I
haven't fired it up yet.

There are zillions of other choices - whatever works is fine. I hadn't
realized how much not having something like ESLint was bugging me...figured
someone else might feel the same.

Oh, set up a local Web server to avoid single-origin hassles from file://
references and to get a more realistic outcome prior to deployment. 4D is a
good choice for this, obviously.

-------
Bonus
-------
Hey, this has nothing to do with any of the above, but check this out:

http://thetruesize.com/

Ever hear "The newest US marine reserve is twice the size of Texas!" Okay,
great. How big is Texas? I have no idea. Oh wait, it's half the size of the
new marine reserve. Hmmm. It's impossible to get a grasp of relative size
unless at least one of the things is something you know...and everyone
knows different places. With "The True Size of..." site, you can find a
place you know, an drag its outline over another place using accurate
projects. I have been wanting this very functionality for years and figure
that some other folks here might enjoy it.
**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ: http://lists.4d.com/faqnug.html
Archive: http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub: mailto:4D_Tech-***@lists.4D.com
*******************************************
Peter Jakobsson
2015-12-18 01:21:32 UTC
Permalink
Thanks very much for taking the time to post those observations and resources David.

Very interesting.

Peter

**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ: http://lists.4d.com/faqnug.html
Archive: http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub: mailto:4D_Tech-***@lists.4D.com
*******************************************************
Ortwin Zillgen
2015-12-18 10:26:20 UTC
Permalink
David

many thanks for your short novel.

Instead of answering with another novel
<http://dddd.mettre.de/wp/svg-state-of-the-art/?inug>



Regards
O r t w i n Z i l l g e n
---------------------------------------------
<mailto:***@mettre.de> <http://dddd.mettre.de/wp/?4713>
<feed://dddd.mettre.de/dddd.xml> <https://twitter.com/Ozett>
member of developer-network <http://www.die4Dwerkstatt.de>

**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ: http://lists.4d.com/faqnug.html
Archive: http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub: mailto:4D_Tech-***@lists.4D.com
*****************************************************
Peter Jakobsson
2015-12-18 16:46:37 UTC
Permalink
Post by Ortwin Zillgen
Instead of answering with another novel
<http://dddd.mettre.de/wp/svg-state-of-the-art/?inug>
SVG does look very powerful.

But what do people use it for ? I can’t see any use for it in business programming.

Warehouse stacking layouts ? CAD/CAM ? Flight control displays for the A350 ? 4D Asteroids ? Interactive Clothes Design Software ? High Tension Switching Displays ?

I’d be interested in hearing what the use cases are for it since it seems to me to fall into the chasm between data entry/reporting (4D forms) and charting (online chart tools).

Regards

Peter


**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ: http://lists.4d.com/faqnug.html
Archive: http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub: mailto:4D_Tech-***@lists.4D.com
**********************************
Jeffrey Kain
2015-12-18 16:56:59 UTC
Permalink
Used it a couple of months ago to print registration badges from 4D. The text had to be printed upside down for proper feeding into the printer, which can't be done in 4D but was a easily handled with SVG.
Post by Peter Jakobsson
Post by Ortwin Zillgen
Instead of answering with another novel
<http://dddd.mettre.de/wp/svg-state-of-the-art/?inug>
SVG does look very powerful.
But what do people use it for ? I can’t see any use for it in business programming.
**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ: http://lists.4d.com/faqnug.html
Archive: http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub: mailto:4D_Tech-***@lists.4D.com
*****************************************
Cannon Smith
2015-12-18 17:06:19 UTC
Permalink
We use it to show a specialized type of graph to feed truck drivers so they can decide whether to increase feed to cattle each morning. Also, to show a second-by-second “playback” on a map of where GPS-enabled feed trucks were throughout the day and what they were doing. A few other small things. I find SVG really useful.

--
Cannon Smith
Synergy Farm Solutions Inc.
Hill Spring, AB Canada
403-626-3236
<***@synergyfarmsolutions.com>
<www.synergyfarmsolutions.com>
Post by Peter Jakobsson
I’d be interested in hearing what the use cases are for it since it seems to me to fall into the chasm between data entry/reporting (4D forms) and charting (online chart tools).
**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ: http://lists.4d.com/faqnug.html
Archive: http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub: mailto:4D_Tech-***@lists.4D.com
******************************************************
Cannon Smith
2015-12-18 17:18:29 UTC
Permalink
Oh, and for customers to make custom “heads up displays” for them to view as they process and treat cattle. It allows them to drag fields, labels, and shapes onto a window, move, resize, rotate each object, change the font and other attributes, etc. Then this customized information is shown to them in a window during a job. All fairly simply with SVG.

--
Cannon Smith
Synergy Farm Solutions Inc.
Hill Spring, AB Canada
403-626-3236
<***@synergyfarmsolutions.com>
<www.synergyfarmsolutions.com>
Post by Cannon Smith
We use it to show a specialized type of graph to feed truck drivers so they can decide whether to increase feed to cattle each morning. Also, to show a second-by-second “playback” on a map of where GPS-enabled feed trucks were throughout the day and what they were doing. A few other small things. I find SVG really useful.
**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ: http://lists.4d.com/faqnug.html
Archive: http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub: mailto:4D_Tech-***@lists.4D.com
*************************************
Peter Jakobsson
2015-12-18 17:31:18 UTC
Permalink
Lots of cool stuff
Wow. I’m very impressed by these applications. Thanks to all for posting. I didn’t realise BubblechartPro was a 4D application - looks a superb piece of work.

I think the closest to the A350 flight control unit at the moment must go to….
a second-by-second “playback” on a map of where GPS-enabled feed trucks were throughout the day and what they were doing
Nice !

Peter

**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ: http://lists.4d.com/faqnug.html
Archive: http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub: mailto:4D_Tech-***@lists.4D.com
**************************************************
Robert McKeever
2015-12-18 17:08:32 UTC
Permalink
Make calendars with clickable day squares.

Scheduling grid where each line is drawn with SVG and placed in a table for display. Some sites have grids drawn daily with 10,000 lines to book over a year in advance. Differing colors for the line background (what type of time {office or surgery}), and the appointments by type and duration.

Custom headers for images coming from the labs.

No end of useful things you can do with it.
Post by Peter Jakobsson
Post by Ortwin Zillgen
Instead of answering with another novel
<http://dddd.mettre.de/wp/svg-state-of-the-art/?inug>
SVG does look very powerful.
But what do people use it for ? I can’t see any use for it in business programming.
Warehouse stacking layouts ? CAD/CAM ? Flight control displays for the A350 ? 4D Asteroids ? Interactive Clothes Design Software ? High Tension Switching Displays ?
I’d be interested in hearing what the use cases are for it since it seems to me to fall into the chasm between data entry/reporting (4D forms) and charting (online chart tools).
Regards
Peter
**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ: http://lists.4d.com/faqnug.html
Archive: http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
**********************************************************************
_________________________________________
Bob McKeever http://www.mswl.com <http://www.mswl.com/>
McKeever's Software Wizardry
Maple Ridge, B.C.
***@mac.com



**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ: http://lists.4d.com/faqnug.html
Archive: http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub: mailto:4D_Tech-***@lists.4D.com
*****************************************************************
George Huhn
2015-12-18 17:24:36 UTC
Permalink
Hi Peter,

I use it for making highly-customizable interactive charts for data visualization and analysis.

For example, all the charts here were created using the SVG component:

http://www.bubblechartpro.com/content/bubble_chart_screenshots.php

Once you learn it, the SVG component makes it easy to link graphic objects to data, so the user can interact with the data using the objects.

Best regards,

George
--
George F. Huhn, M.S.E,
President, Data Machines, Inc.
Blog: http://blog.datamachines.com
Follow us at http://twitter.com/DataMachinesCom
302-529-1193
Data Machines, Inc., 122 Broadbent Road, Wilmington, DE 19810
Post by Peter Jakobsson
SVG does look very powerful.
But what do people use it for ? I can’t see any use for it in business programming.
**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ: http://lists.4d.com/faqnug.html
Archive: http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub: mailto:4D_Tech-***@lists.4D.com
**************
John Baughman
2015-12-18 17:53:24 UTC
Permalink
I use SVG to create orthodontic treatment cards. These are graphic representations of each patient’s mouth/teeth depicting the wires/elastics applied. The technicians choose type, size, and color of the elastic being applied then clicks on each tooth where the elastic is attached. Using SVG the elastic is drawn between the teeth as the tech clicks.

SVG and JSON made it fairly easy exercise to extend this functionality to an iPad app used at chair side. The treatment cards can be created, viewed, and edited on both the iPad and in 4D, all managed by the same set of 4D methods utilizing SVG..


John
Post by Peter Jakobsson
Post by Ortwin Zillgen
Instead of answering with another novel
<http://dddd.mettre.de/wp/svg-state-of-the-art/?inug>
SVG does look very powerful.
But what do people use it for ? I can’t see any use for it in business programming.
Warehouse stacking layouts ? CAD/CAM ? Flight control displays for the A350 ? 4D Asteroids ? Interactive Clothes Design Software ? High Tension Switching Displays ?
I’d be interested in hearing what the use cases are for it since it seems to me to fall into the chasm between data entry/reporting (4D forms) and charting (online chart tools).
Regards
Peter
**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ: http://lists.4d.com/faqnug.html
Archive: http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
**********************************************************************
**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ: http://lists.4d.com/faqnug.html
Archive: http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub: mailto:4D_Tech-***@lists.4D.com
******************
Jim Dorrance
2015-12-18 18:40:11 UTC
Permalink
I did an SVD charting component for a European distributer. The big guy
with the cigar who does media buying can visualise past deals when
negotiating
buying media space by telephone.
Post by Peter Jakobsson
Post by Ortwin Zillgen
Instead of answering with another novel
<http://dddd.mettre.de/wp/svg-state-of-the-art/?inug>
SVG does look very powerful.
But what do people use it for ? I can’t see any use for it in business programming.
Warehouse stacking layouts ? CAD/CAM ? Flight control displays for the
A350 ? 4D Asteroids ? Interactive Clothes Design Software ? High Tension
Switching Displays ?
I’d be interested in hearing what the use cases are for it since it seems
to me to fall into the chasm between data entry/reporting (4D forms) and
charting (online chart tools).
Regards
Peter
**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ: http://lists.4d.com/faqnug.html
Archive: http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
**********************************************************************
--
Jim Dorrance
***@gmail.com
***@dorrance.eu
www.4d.dorrance.eu

PS: If you know of anyone that needs an experienced 4D programmer to add
energy and experience to their team, please let me know. I have
experience in many areas. Reasonable rates. Remote or Paris only.
**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ: http://lists.4d.com/faqnug.html
Archive: http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub: mailto:4D_Tech-***@lists.4D.com
********************************************
David Adams
2015-12-18 21:23:47 UTC
Permalink
Ortwin, thanks for all of the links to your write ups - very interesting. I
now have a better idea of what's possible within 4D for SVG display and
production.
**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ: http://lists.4d.com/faqnug.html
Archive: http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub: mailto:4D_Tech-***@lists.4D.com
*****************************************************************
Tom DeMeo
2015-12-18 18:07:52 UTC
Permalink
Hi,


One of the most compelling things about SVG on the web for me was that it gives you tools to implement drag and drop.

I wrote a toolkit to use SVG to layout HTML elements in the browser. (caveat: I did use Raphael, not D3)


SVGLayout Demo <http://www.svglayout.com/>


There is a floating button in the upper right hand corner (with an SL logo). Click on that to turn on layout mode, if you want to try it.



Tom
**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ: http://lists.4d.com/faqnug.html
Archive: http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub: mailto:4D_Tech-***@lists.4D.com
******************************
Keith Culotta
2015-12-18 18:29:24 UTC
Permalink
Hi,

I used SVG replace some of the drawing functions of 4D Draw for a customer. Fortunately, they used a minimum of features. The SVG engine is really fast when it comes to redrawing a picture.
It's also been useful to let the user design their own input forms.

Keith - CDI
Post by Tom DeMeo
Hi,
One of the most compelling things about SVG on the web for me was that it gives you tools to implement drag and drop.
I wrote a toolkit to use SVG to layout HTML elements in the browser. (caveat: I did use Raphael, not D3)
SVGLayout Demo <http://www.svglayout.com/>
There is a floating button in the upper right hand corner (with an SL logo). Click on that to turn on layout mode, if you want to try it.
Tom
**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ: http://lists.4d.com/faqnug.html
Archive: http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
**********************************************************************
**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ: http://lists.4d.com/faqnug.html
Archive: http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub: mailto:4D_Tech-***@lists.4D.com
***************************************************************
David Adams
2015-12-18 21:28:57 UTC
Permalink
// -- TSV (etc.) is a pain because everything is imported as a string.
There's no
// schema or meta-data to guide D3 in how to interpret the data. You can do
a
// post conversion within your JS, like this fragment:
//
// d3.tsv('./bubble_data/locations.tsv', function(data) {
// /* Convert strings to numbers. If you use JSON instead of TSV, you don't
// /need this step. */
// data.forEach(function(d){ d['Total Species'] = +d['Total Species'];
// d['Shared Species'] = +d['Shared Species']; d['Shared Percent'] =
// +d['Shared Percent']; d['Endemic Percent'] = +d['Endemic Percent'];
// d['Square KMs'] = +d['Square KMs'];
// })
// }
// Is that a good example? I don't know...it might be a terrible example,
but it
// does show the kind of code to look for. Note that there's a solid
argument
// saying that this make the JS and the source data a bit to intimate. Nice
to
// have clean data with type definitions inside of it that is simply passed
// through to D3. If you feel this way, use JSON.

Writing things down can be a good way to make them clearer...An obvious
solution to the problem of untyped elements in tsv imports is to add your
own map. Say:

tsvImportWithMap('./bubble_data/locations.tsv',
'./bubble_data/type_map.json' {....} returns adjusted data.

The type_map.json file would contain the names of elements that need
conversion, what conversion is needed and, depending on what you need, a
format description or method hint. A lot of cases take nothing more than
date and number conversions so it's not a complicatd task.
**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ: http://lists.4d.com/faqnug.html
Archive: http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub: mailto:4D_Tech-***@lists.4D.com
************************************************
David Adams
2015-12-18 21:32:32 UTC
Permalink
Thanks to everyone for all of the cool SVG examples. Impressive and
inspiring.

One thing I really like about using D3 is that all 4D needs to do is spit
out properly formatted data. No SVG. I did the SVG stuff by hand because I
wanted to get my hands dirty with it. D3 is pretty hard to beat...and there
are a lot of cases where it's easy to imagine 4D is an ideal source for
generating the data. For anyone that needs a different image format,
PhantomJS converts SVG files just fine. The 4D code to convert an SVG to
PDF or PNG (etc.) is just a few lines. I posted a demo some months back and
reused the code recently. I didn't remember how it worked in detail...I
just saw how to call it and it worked. So, if you don't know PhantomJS - or
JavaScript - it might not matter.

Check out some of the D3 examples and see what's possible. There's tons of
support for prertty exotic and/or hard to figure out charts,
animations/transitions, hover behaviors, etc.
**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ: http://lists.4d.com/faqnug.html
Archive: http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub: mailto:4D_Tech-***@lists.4D.com
***************************************************************
David Adams
2015-12-18 22:01:43 UTC
Permalink
For those of you doing Web-facing deployments of 4D data that are
interested in graphing with D3, I thought I'd provide a direct link to one
of the more relevant parts of the API docs:

https://github.com/mbostock/d3/wiki/Requests

This is the D3 version of XMLHttpRequest. Imagine you've got a dashboard
with various elements and you want to pull some live data (appointments,
gauge readings, latest poll figures, etc.) out of 4D. There you go....all
you need is a tiny REST API and away you go.
**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ: http://lists.4d.com/faqnug.html
Archive: http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub: mailto:4D_Tech-***@lists.4D.com
*************
Olivier Deschanels
2015-12-19 07:56:12 UTC
Permalink
Hello,

For french speakers (or Google translator users), you can read my blog article on this subject :
http://www.4d.com/fr/blog/bubblegraphe.html

Regards,

Olivier
Post by David Adams
For those of you doing Web-facing deployments of 4D data that are
interested in graphing with D3, I thought I'd provide a direct link to one
https://github.com/mbostock/d3/wiki/Requests
This is the D3 version of XMLHttpRequest. Imagine you've got a dashboard
with various elements and you want to pull some live data (appointments,
gauge readings, latest poll figures, etc.) out of 4D. There you go....all
you need is a tiny REST API and away you go.
**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ: http://lists.4d.com/faqnug.html
Archive: http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
**********************************************************************
**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ: http://lists.4d.com/faqnug.html
Archive: http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub: mailto:4D_Tech-***@lists.4D.com
*************************
David Adams
2015-12-19 09:32:28 UTC
Permalink
Olivier,

Great article! (Google translate looked to do a very good job here.) I'd
encourage people to check out Olivier's post as it shows an example of 4D
and D3 used together effectively.
**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ: http://lists.4d.com/faqnug.html
Archive: http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub: mailto:4D_Tech-***@lists.4D.com
**********************************
Olivier Deschanels
2015-12-19 13:11:22 UTC
Permalink
Thanks David,
A recommandation from you is like a Christmas present for me :-)

d3.js is a great tool and is so easy to include in 4D projects through a web area. I add some example (like this one in my article) in almost all the prototypes I build with partners when we approach a new customer and in all case it’s a great success even if it’s not the central part of the project. In fact as it’s flashy and new for most of the people, this keeps the attention.

Regards,

Olivier
Post by David Adams
Olivier,
Great article! (Google translate looked to do a very good job here.) I'd
encourage people to check out Olivier's post as it shows an example of 4D
and D3 used together effectively.
**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ: http://lists.4d.com/faqnug.html
Archive: http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
**********************************************************************
**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ: http://lists.4d.com/faqnug.html
Archive: http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub: mailto:4D_Tech-***@lists.4D.com
*************************
Kirk Brooks
2015-12-26 20:50:53 UTC
Permalink
Olivier,
When I attempt to execute D3 in a web area and make a call, any call, to
d3.json I get an error that seems to track back to this note
<https://github.com/mbostock/d3/wiki#using> on D3.org:

When developing locally, note that your browser may enforce strict
permissions for reading files out of the local file system. If you use
d3.xhr locally (including d3.json et al.), you must have a local web
server.


​The obvious solution is to simply run the page through the 4D web server
but anyone on Windows runs into this:

Web Area and Web server conflict (Windows)
<http://doc.4d.com/4Dv15/4D/15.1/Programmed-management-of-Web-Areas.300-2685006.en.html>


Under Windows, it is not recommended to access, via a Web area, the Web
server of the 4D application containing the area because this configuration
could lead to a conflict that freezes the application. Of course, a remote
4D can access the Web server of 4D Server, but not its own Web server.

​Plus it would be pretty expensive to require a user of a single user db to
get a web server license just to see D3 in a web area.

I'm wondering if anyone else has come up with a solution to this problem?

To recap the specific problem is this:

Single user db, open a form with a web area and load a page using D3 and
attempt to import a json using d3.json.


​FYI - I also tried embedding the json in the page itself and simply
accessing that to no avail.

On Fri, Dec 18, 2015 at 11:56 PM, Olivier Deschanels <
Post by Olivier Deschanels
​F​
or french speakers (or Google translator users), you can read my blog
http://www.4d.com/fr/blog/bubblegraphe.html
--
Kirk Brooks
San Francisco, CA
=======================
**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ: http://lists.4d.com/faqnug.html
Archive: http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub: mailto:4D_Tech-***@lists.4D.com
*********************************
David Adams
2015-12-26 21:27:52 UTC
Permalink
I haven't tried out D3 in a WebArea and am not testing any of this on
Windows...so I may not have anything useful to say...but I'll offer a few
points.


* You make a good point about a stand-alone solution and WebAreas. If they
don't work smoothly with the 4D Web server, than some JS libraries aren't
going to make sense. If so, bummer.

* Yeah, you really need a local Web server to work with D3. That needn't
cost anything and needn't be 4D. I'm using Apache on OS X and it's fine.
Granted, that's fine for many sorts of development and deployment, but not
so much if you're going for a simple stand-alone 4D solution.

* See if you can get the page to work in a local browser to make sure that
there's no problem there. I'm suspicious because you said that putting the
JSON in the page itself doesn't work. Try this in a browser and look at the
JS console in whatever version of developer tools you've got in your
browser. (Chrome's version is decent enough.)

* Some browsers (but only some) let you avoid single-origin hassles when
you include a JSONP ("JSON with Padding") file. You don't import it with a
parsing commands, you just declare it as a script:

<script src="../assets/my.wonderful.data.jsonp" type="text/javascript"
</script>
Your JSONP file has a variable declaration and the relevant data. This
variable is then global to your page and can be used throughout. This is a
nice way to get started, but it's not ideal to have everything up in the
global space, IMO. But I would say that ;-) I prefer
**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ: http://lists.4d.com/faqnug.html
Archive: http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub: mailto:4D_Tech-***@lists.4D.com
***************
Kirk Brooks
2015-12-26 21:58:36 UTC
Permalink
Hey David,
Thanks for the suggestions. After some looking around it seems the simplest
solution to this is SimpleHTTPServer that's part of Python. We've got it
built into Mac OS and it sounds like newer versions of Windows have Python
installed already as well.

I invoked it using iTerm on port 8888. First I navigated to the folder
where I've got the web pages in question then ran:

python -m SimpleHTTPServer 8888


​which I got from here <https://github.com/mbostock/d3/wiki#using>. It
seems whatever dir you are in when you invoke the command becomes the root
for that instance of the server. I have not tried running multiple servers
on different ports but it seems perfectly doable though the opportunities
to truly get confused abound.

Anyway, this works great and is dead simple. I could open the pages using
any browser I wanted including Web Areas. In practice I think I will have
various templates for different graphs I want to show which will reference
some data doc. The data docs will be spit out by 4D. This way I can
manipulate the js and D3 completely independent of 4D and manipulate the
data independent of the javascript.

Now I'm investigating how to start - and stop - the server using LEP so
it's all transparent to the user. At the moment I'm wondering how to write
the equivalent of pressing Control C in the terminal...
Post by David Adams
I haven't tried out D3 in a WebArea and am not testing any of this on
Windows...so I may not have anything useful to say...but I'll offer a few
points.
* You make a good point about a stand-alone solution and WebAreas. If they
don't work smoothly with the 4D Web server, than some JS libraries aren't
going to make sense. If so, bummer.
* Yeah, you really need a local Web server to work with D3. That needn't
cost anything and needn't be 4D. I'm using Apache on OS X and it's fine.
Granted, that's fine for many sorts of development and deployment, but not
so much if you're going for a simple stand-alone 4D solution.
--
Kirk Brooks
San Francisco, CA
=======================
**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ: http://lists.4d.com/faqnug.html
Archive: http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub: mailto:4D_Tech-***@lists.4D.com
*******************************
David Adams
2015-12-27 05:48:40 UTC
Permalink
wrote: Hey
After some looking around it seems the simplest solution to this is
SimpleHTTPServer
that's part of Python....Anyway, this works great and is dead simple.
Nice one! Thanks for posting it to the list for the good of the archives.
In practice I think I will have various
templates for different graphs I want to show which will reference some
data
doc. The data docs will be spit out by 4D. This way I can manipulate the
js
and D3 completely independent of 4D and manipulate the data independent
of the
javascript.
That sounds 100% compatible with the philosophy of D3. Ideally, you get to
reuse a single data source for multiple graphs...that's also helps make
those sweet transitions efficient. For those following along, Mike Bostock
is the primary author of D3. Many of his examples show various graphs drawn
from data in a simple flat file. D3 automatically handles stacks of formats
and you can also implement custom imports. A potential gotcha when coming
from a 4D mindset is that all of D3's import calls are asynchronous.
(That's probably why a lot of examples work with small, embedded data
sets.) When working with external files (local or remote), you can set the
import call to run a callback once the file is loaded. This is really easy
to do once you know you need to do it. This routine calls for a file to be
loaded at the location in dataUrl ($1) and, once the file is loaded, to
call handleMap with the data.

function distributionMap (dataUrl) {
d3.json (dataUrl, function (error, data) {
return handleMap (data);
});
}

the handleMap gets the data once its loaded in series - you can call it
whatever you like.
function handleMap (series) {
.... "series" above is "data" from the first

So, it's over in the callback that you run the real code for graphing. If
you try doing it right in the routine that calls d3.json (d3.tsv, etc.),
you *sometimes* end up with an undefined object - it just depends how
quickly the data loads. Nasty but avoidable. If you need multiple files to
load before starting a script, there are a few strategies. Actually, there
are dozens. I don't know what bright spark decided to make JS purely
asynchronous...but I really do wonder what they were thinking. I mean, I
get it up to a point...but only to a point. Anyway, there are zillions of
options, libraries, browser features, and
future-language-versions-that-you-can-use-now-via-Babel...but I went with
Queue.js:

https://github.com/mbostock/queue

Yup, it's by the same guy as wrote D3. He creates a synchronous queue with
a freakishly small script. Here's a loader that uses a queue to load three
files before calling the main handler:

function distributionMapWithEndemicBubbles (dataUrl, bubblesUrl,
optionsURL) {
queue()
.defer(d3.json, dataUrl)
.defer(d3.json, bubblesUrl)
.defer(d3.json, optionsURL)
.await(handleMapX);
}

That's it. In the example above, I'm using a topoJSON (also by Mike
Bostock) map and shading each country depending on how many species are
there. So, Brazil is darker than the USA and Australia is lighter than
Indonesia. Anyway, that file gets used in a stand-alone map and then reused
as shown above. In the second example, countries with endemic (unique to
that place) get a bubble sized based on the number of endemics. So,
Australia gets a huge bubble and Italy doesn't get any bubble at all. Map
graphs of various sorts are pretty great. If you need anything like that,
check out DataMaps:

https://github.com/markmarkoh/datamaps

DataMaps is pretty awesome and you can get some great results very, very
easily. The author (Mark DiMarco) has included heaps of topoJSON map files
in the distribution. Most examples use the world with each country a region
or the USA with each state a region. Every country has its own individual
file in the distribution too. This is the best page to start from:

https://github.com/markmarkoh/datamaps

Some of the docs 404 and the docs aren't comprehensive...but this library
is widely used and the author is very responsive to questions. (I asked a
dumb question and got a polite answer very quickly. I'm not worthy.) Oh,
and you'll see that this library depends on D3 and topoJSON, both by Mike
Bostock as mentioned. D3 is a platform.

Getting back to reusing files, apart from filter(), check out nest() and
rollup(). There's a lot you can do once you've spit the data out.

Speaking of spitting the data out, D3 has some facilities for reformatting
data and JS has the rest...sometimes you need an array and you've got an
object, etc. In our case, we can use 4D to spit things out in whatever
format is needed. Speaking of 4D, it's proving to be a fantastic platform
for me in this case. I can use it to pull in and clean data from the Web
automatically, which makes all of this possible. Then I've got about a
dozen main files that need to join across up to about five tables over and
over very regularly to calculate things. (I've denormalized in one place
and am glad of it, but otherwise this is a pretty purely 3NF+ structure.)
Fortunately, I've got some generic joining code - but that's easy in 4D.
I'll tell you what's great in 4D: sets. Sets are awesome. In some cases,
you can radically optimize finding and counting things by iterating with
INTERSECT. Kudos to 4D. On the output side, 4D's also great. This stuff is
all bolted onto my main code base so it's in V13 for now (although I'd be
happy to move to 14 or 15...I just haven't taken the time and I've got
clients depending on the V13 version working...so.) Anyway, I'm writing my
own text outputers, using NTK's JSON commands, and PROCESS HTML TAGS -
whatever gets the job done.

For anyone still interested in D3, there's something important to know
about and appreciate...but the terminology might be confusing or misleading
to we 4D people at first. Look here at "Thinking with Joins":

http://bost.ocks.org/mike/join/

Why is this feature/set of concepts called join? I've not a clue. But
what's underneath it all is excessively good. Part of what D3 offers are
transitions that are respectful of the data. So, you can have a bar chart,
click on a different category/period and see the bars move and reorder
visually. This isn't just an animation gimmick, supposedly it can help
someone track an element move from one spot to another. But that's a lot of
rendering. Lots of rendering = slow. SVG has a reputation for being
slow...and D3 is all about SVG, CSS & JS. But it's not slow. Go check out
examples. Not slow. How? Well, a couple of things, but first off nodes are
marked as being in one of three states "enter", "update", and "exit." I
would translate these as "new", "modified", and "deleted." Perhaps
enter-update-exit come from animation? Anyway, the renderer only has to
clear "exit" items, update "update" items (say, adjust text or the x
position), and add "enter" items. Less rendering. I think that you also get
events you can hang code off of...but I haven't tried that yet.

The more important point on that page is about D3's declarative nature. One
thing that's pretty striking in D3 examples is how very little code there
is. (Not mine, I don't know what I'm going yet...I mean the examples from
grown-ups.) The only other time I tried working with a declarative language
was with XSLT...which I found quite hard. JavaScript is not a declarative
language but D3 has a very declarative style. Instead of saying how to do
something, you say what you want done. So, instead of iterating over a
bunch of data positioning and formatting it, you hand an array of data to
D3 and tell it what you want to have happen. This whole paragraph is a bad
summary of a well-written piece. Mike Bostock can also write well. What
can't he do? Maybe he sucks at cooking or something.

Well, that was another big dump of random details, but perhaps they'll be
of use somewhere down the line to someone. D3 is just an amazing piece of
work and I highly recommend it to anyone that's got a need for data-based
visualizations and a bit of time. You can get a remarkable amount done
without knowing a lot of JavaScript by bolting things together. That's no
way to live long-term, but can be enough to get started and then you get
hooked...Oh, and if you need rendered versions of graphs, PahntomJS does
work just fine.

P.S. Bonus URL:

http://colorbrewer2.org/

Picking a cognitively correct color scheme for a map is, as it turns out,
hard. If you look for decent color arrays, you'll run into the site above
pretty quickly. But, to save you the time, the link is above.
**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ: http://lists.4d.com/faqnug.html
Archive: http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub: mailto:4D_Tech-***@lists.4D.com
********************************************
Kirk Brooks
2015-12-27 19:23:54 UTC
Permalink
David,
Another great post - thank you. This is the sort of stuff that is
interesting reading right now and really useful to have archived and come
back to.
Post by Kirk Brooks
After some looking around it seems the simplest solution to this is
​ ​
SimpleHTTPServer
that's part of Python....Anyway, this works great and is dead simple.
Nice one! Thanks for posting it to the list for the good of the archives.
​I'm going to update this with the actual code snippet to make it work from
within 4D for anyone coming in later.

​ // path to the folder you want to be the 'root' for the http server
$path:=Get 4D folder(Current resources folder)+"html"+Folder separator
SET ENVIRONMENT VARIABLE("_4D_OPTION_CURRENT_DIRECTORY";$path)
SET ENVIRONMENT VARIABLE("_4D_OPTION_HIDE_CONSOLE";"true") // hide the
console
SET ENVIRONMENT VARIABLE("_4D_OPTION_BLOCKING_EXTERNAL_PROCESS";"false")
// without this the 4D process keeps running

$command:="python -m SimpleHTTPServer 8888"

LAUNCH EXTERNAL PROCESS($command)


​This launches the http server on port 8888. It runs as a shell command so
it is independent of 4D and KEEPS RUNNING after you quit 4D. I'll post some
code to stop it when you shut down 4D once I get around to it. Probably not
an issue but would be better to clean it up.

Also LEP doesn't start up lickety split so if your code does something like:

spit out a data doc

start the http server

open a web area

you get a blank white form. Rumor has it around here that LEP takes at
least 250 ms to start up so you need to either start the http server as
part of the db start up or delay the process making the first call to use
it.

Speaking of spitting the data out, D3 has some facilities for reformatting
Post by Kirk Brooks
data and JS has the rest...sometimes you need an array and you've got an
object, etc. In our case, we can use 4D to spit things out in whatever
format is needed. Speaking of 4D, it's proving to be a fantastic platform
for me in this case. I can use it to pull in and clean data from the Web
automatically, which makes all of this possible. Then I've got about a
dozen main files that need to join across up to about five tables over and
over very regularly to calculate things. (I've denormalized in one place
and am glad of it, but otherwise this is a pretty purely 3NF+ structure.)
Fortunately, I've got some generic joining code - but that's easy in 4D.
I'll tell you what's great in 4D: sets. Sets are awesome. In some cases,
you can radically optimize finding and counting things by iterating with
INTERSECT. Kudos to 4D. On the output side, 4D's also great. This stuff is
all bolted onto my main code base so it's in V13 for now (although I'd be
happy to move to 14 or 15...I just haven't taken the time and I've got
clients depending on the V13 version working...so.) Anyway, I'm writing my
own text outputers, using NTK's JSON commands, and PROCESS HTML TAGS -
whatever gets the job done.
​I agree. When you start using v15 you will really, really like the ability
to manipulate JSON natively even if somewhat idiosyncratically. And I think
the real sleeper feature in v15 right now is your ability to store a JSON
in a 4D object field and query on it. I haven't done this a lot or on any
big datasets but this is a very cool feature. It essentially allows one to
build a 4D database that mimics a noSQL database (like Mongo) without
sacrificing a relational framework. Social computing really likes noSQL and
Mongo a lot. It is really terrible for things like accounting, though. You
know - like getting paid. So there have to be split systems to handle that.


Looking at it from another point of view there's also the issues that come
about when you want to look for relationships. NoSQL is all about
'de-normalizing' the data into discrete objects particular to whatever you
are working with. This is fine for things being viewed by a human but not
so efficient if you want to find things procedurally. Plus loosing the
whole 'write once' benefit. I'm not sure 4D can scale well enough to really
be a contender for places where this is an issue but it seems to me it's a
terrific advantage right now because it's unmatched by anything else with
the various tools you outlined.
Post by Kirk Brooks
Mike Bostock can also write well. What
​ ​
can't he do? Maybe he sucks at cooking or something.
​Yeah - that's it. 4D people are better at cooking - right Balinder?​
--
Kirk Brooks
San Francisco, CA
=======================
**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ: http://lists.4d.com/faqnug.html
Archive: http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub: mailto:4D_Tech-***@lists.4D.com
*************************
Bertrand SOUBEYRAND
2017-01-14 14:23:06 UTC
Permalink
Post by Olivier Deschanels
David,
Another great post - thank you. This is the sort of stuff that is
interesting reading right now and really useful to have archived and come
back to.
I push up this thread with:

http://blog.octo.com/en/d3-js-transitions-killed-my-cpu-a-d3-js-pixi-js-comparison/


Bertrand SOUBEYRAND
---
La Soub Compagnie
33 bd de la Liberté - 13001 Marseille
Bureau : +33 4 84 25 21 99
http://www.soubeyrand-4d-developer.eu <http://www.soubeyrand-4d-developer.eu/>

Belgique : +32 87 84 00 88




**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ: http://lists.4d.com/faqnug.html
Archive: http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub: mailto:4D_Tech-***@lists.4D.com
*********************************************
David Adams
2017-01-14 22:29:19 UTC
Permalink
Bertrand,

Since you brought it up....

That was a pretty interesting piece, but they very unfortunately went to a
great deal of work and missed out on a lot of existing information. Here's
a better post on the same subject:

http://www.visualcinnamon.com/2015/11/learnings-from-a-d3-js-addict-on-starting-with-canvas.html

For those of you following along at home, a recap:

* If what you want are Web-based business charts, pay for HighCharts and
move on. It just looks fantastic and very easy to use. D3 is...something
else. If you need to invent a new chart type, go with D3.

* D3 binds data to DOM elements in a browser. It's SVG. That's the default.
* Browsers render SVG using the CPU (mostly, if not entirely.) That tops
out. People with substantial data sets where they need to render thousands
of moving objects simultaneously hit a performance wall.

* The big problem is the big DOM. Fact of life.

* But SVG is awesome! Transitions, styling, native objects, vectors.
Vectors! So good. But you just hit a wall at some point. Not a D3 specific
issue, but an issue that D3 users run into, if they have huge data sets
with complex rendering. (My data sets aren't that size, but some people's
sets are.)

Enter HTML5 and the Canvas object. It is a *bitmap* rendering space, not a
vector space. Eh? It's pushed out to the GPU, so you can get *massive*
processing power for "free." This is exactly why I said the other day
(roughly): "Cores? Who cares about cores. The cool kids are pushing stuff
to the GPU." You can render ginormous data sets to a bitmap this way and it
works great. These days, it's not that common to optimize something by
pushing it to a different bit of hardware, but that's the case here.

But wait..bitmap...so how do you get events? Transitions? You don't.
They're bitmaps - static. So you have to put more work into simulating
those features. To get transitions (frame animation), you have to renderer
the scene over, and over, and over very quickly. So you end up doing vastly
more work, but the result executes more quickly because of all of the GPU
processing. That's the point behind taking advantage of WebGL. (The post
Bernard listed doesn't seem to explain this - but that's the reason to push
things to WebGL, it's GPU optimized for the bitmaps.)

Okay, we've got animation back. What about events? I mean when you click on
a "circle" or something, there's no circle object there - it's just a pixel
like all of the others. I've seen two approaches. One is that you overlay a
Voronoi diagram:

https://en.wikipedia.org/wiki/Voronoi_diagram

Yeah, like I know what any of that means ;-) There are tons of D3 examples
of overlaying Voronoi diagrams for various reasons.

The other technique (and the one I'll guess that the Pixi library mentioned
uses) is to take advantage of a spare bit of information in the pixel You
can generate a unique 'color index' in a non-display area so that each
pixel has its own unique ID. Kind of cool and very efficient. Then you need
to have a mapping system to translate which "object" that pixel is a part
of. Click-get color index-lookup index in map-figure out if the click is in
an 'object'-run appropriate event handling code.

I haven't implemented either of these techniques. I managed to make it up
to the first day of 4D's very nice world tour last year and JPR showed a
renderer in 4D that had pixel-level responsiveness for millions of pixels.
I assumed he was using a technique similar-if-not-identical-to the second
one. (In which case, moving it to a WebArea + JS + Canvas would get you GPU
acceleration.) I couldn't come to the second day, so I never did find out
if that's how he was doing it!

This is all off the top of my head, first thing in the morning...and,
again, I haven't implemented either of these things. Why, in fact, do I
even know about all of this? Well, when I learn a new tool, I like to poke
around at the edges a lot and see what sort of problems people run into.
That way, if you ever do run into it, you know how other people have
already solved the problem. The idea of an overcrowded DOM is pretty
obvious and easy to run into at scale, so it pops up frequently. I'm
surprised that the folks in the blob post that Bertrand mentioned had to
"reinvent the car" on this one.

Oh, I should also add that Mike Bostock (Mr. D3) is well aware of this
issue and has been for years. He's had lots of posts, proof-of-concept code
for working with Canvas and such. In fact, version 4 of D3 (out for several
months), adds support for abstracting the rendering engine. Meaning, you
can render to SVG or Canvas. At least for some of the primitives. So,
targeting Canvas - for the very reasons mentioned in the post Bertrand
listed - is a core feature of the shipping version of D3 and support will
only increase over time. He's got a bunch of blog posts about it but I
don't have them to hand.

Unfortunately, I haven't been able to work on D3 for about 6 months or
since the V4 release. I've got a roadmap that takes me back to D3, but
there's a few months of prep work before I get there. Looking forward to.
The thing is just amazing. It's a turning point in data visualization, no
joke. Super big deal.
**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ: http://lists.4d.com/faqnug.html
Archive: http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub: mailto:4D_Tech-***@lists.4D.com
********************************
Balinder Walia
2017-01-14 22:41:46 UTC
Permalink
Hi David,

Great to see this post about D3.js _ I am a fan of D3.js :-)

I have been using D3 based JS lib called plotly https://plot.ly/javascript/ to
visualise JSON data from 4D/MongoDB/CSVs/JSON in few projects now. Here are
a few prototypes I produced in node.js: http://plotly.tenthmatrix.co.uk/
the great things about this tool is it is super quick to just start
building charts. Very less coding required.
http://plotly.tenthmatrix.co.uk/test7 is dynamically fetching changed data
from server.

I have recently added plotly.js in my upcoming CRM app release - I have
also used these charts in 4D desktop and our JS Team India has build chart
reporting system for Cloud SAAS business apps with great results.

I will be releasing some examples working with code on GitHub soon. Stay
tuned :-)

Cheers / Balinder
Post by David Adams
Bertrand,
Since you brought it up....
That was a pretty interesting piece, but they very unfortunately went to a
great deal of work and missed out on a lot of existing information. Here's
http://www.visualcinnamon.com/2015/11/learnings-from-a-d3-
js-addict-on-starting-with-canvas.html
* If what you want are Web-based business charts, pay for HighCharts and
move on. It just looks fantastic and very easy to use. D3 is...something
else. If you need to invent a new chart type, go with D3.
* D3 binds data to DOM elements in a browser. It's SVG. That's the default.
* Browsers render SVG using the CPU (mostly, if not entirely.) That tops
out. People with substantial data sets where they need to render thousands
of moving objects simultaneously hit a performance wall.
* The big problem is the big DOM. Fact of life.
* But SVG is awesome! Transitions, styling, native objects, vectors.
Vectors! So good. But you just hit a wall at some point. Not a D3 specific
issue, but an issue that D3 users run into, if they have huge data sets
with complex rendering. (My data sets aren't that size, but some people's
sets are.)
Enter HTML5 and the Canvas object. It is a *bitmap* rendering space, not a
vector space. Eh? It's pushed out to the GPU, so you can get *massive*
processing power for "free." This is exactly why I said the other day
(roughly): "Cores? Who cares about cores. The cool kids are pushing stuff
to the GPU." You can render ginormous data sets to a bitmap this way and it
works great. These days, it's not that common to optimize something by
pushing it to a different bit of hardware, but that's the case here.
But wait..bitmap...so how do you get events? Transitions? You don't.
They're bitmaps - static. So you have to put more work into simulating
those features. To get transitions (frame animation), you have to renderer
the scene over, and over, and over very quickly. So you end up doing vastly
more work, but the result executes more quickly because of all of the GPU
processing. That's the point behind taking advantage of WebGL. (The post
Bernard listed doesn't seem to explain this - but that's the reason to push
things to WebGL, it's GPU optimized for the bitmaps.)
Okay, we've got animation back. What about events? I mean when you click on
a "circle" or something, there's no circle object there - it's just a pixel
like all of the others. I've seen two approaches. One is that you overlay a
https://en.wikipedia.org/wiki/Voronoi_diagram
Yeah, like I know what any of that means ;-) There are tons of D3 examples
of overlaying Voronoi diagrams for various reasons.
The other technique (and the one I'll guess that the Pixi library mentioned
uses) is to take advantage of a spare bit of information in the pixel You
can generate a unique 'color index' in a non-display area so that each
pixel has its own unique ID. Kind of cool and very efficient. Then you need
to have a mapping system to translate which "object" that pixel is a part
of. Click-get color index-lookup index in map-figure out if the click is in
an 'object'-run appropriate event handling code.
I haven't implemented either of these techniques. I managed to make it up
to the first day of 4D's very nice world tour last year and JPR showed a
renderer in 4D that had pixel-level responsiveness for millions of pixels.
I assumed he was using a technique similar-if-not-identical-to the second
one. (In which case, moving it to a WebArea + JS + Canvas would get you GPU
acceleration.) I couldn't come to the second day, so I never did find out
if that's how he was doing it!
This is all off the top of my head, first thing in the morning...and,
again, I haven't implemented either of these things. Why, in fact, do I
even know about all of this? Well, when I learn a new tool, I like to poke
around at the edges a lot and see what sort of problems people run into.
That way, if you ever do run into it, you know how other people have
already solved the problem. The idea of an overcrowded DOM is pretty
obvious and easy to run into at scale, so it pops up frequently. I'm
surprised that the folks in the blob post that Bertrand mentioned had to
"reinvent the car" on this one.
Oh, I should also add that Mike Bostock (Mr. D3) is well aware of this
issue and has been for years. He's had lots of posts, proof-of-concept code
for working with Canvas and such. In fact, version 4 of D3 (out for several
months), adds support for abstracting the rendering engine. Meaning, you
can render to SVG or Canvas. At least for some of the primitives. So,
targeting Canvas - for the very reasons mentioned in the post Bertrand
listed - is a core feature of the shipping version of D3 and support will
only increase over time. He's got a bunch of blog posts about it but I
don't have them to hand.
Unfortunately, I haven't been able to work on D3 for about 6 months or
since the V4 release. I've got a roadmap that takes me back to D3, but
there's a few months of prep work before I get there. Looking forward to.
The thing is just amazing. It's a turning point in data visualization, no
joke. Super big deal.
**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ: http://lists.4d.com/faqnug.html
Archive: http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
**********************************************************************
**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ: http://lists.4d.com/faqnug.html
Archive: http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub: mailto:4D_Tech-***@lists.4D.com
*************
David Adams
2017-01-14 23:15:37 UTC
Permalink
Balindar,

Your stuff looks sharp! I'm excited to hear that you'll be releasing some
of your examples.

I'm embarassed to say that I didn't know that Plot.ly uses D3 as it's SVG
generating engine. I've had another look at the Plot.ly Web site, and it
looks like a pretty awesome and well thought out product.

Any chance you'll be doing a Summit presentation on Plot.ly?
**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ: http://lists.4d.com/faqnug.html
Archive: http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub: mailto:4D_Tech-***@lists.4D.com
*************************
Balinder Walia
2017-01-15 00:10:46 UTC
Permalink
Thanks David,

Yeah sure It should be possible for me to do *Plot.ly and 4D* presentation
with working demo.

Cheers
Post by David Adams
Balindar,
Your stuff looks sharp! I'm excited to hear that you'll be releasing some
of your examples.
I'm embarassed to say that I didn't know that Plot.ly uses D3 as it's SVG
generating engine. I've had another look at the Plot.ly Web site, and it
looks like a pretty awesome and well thought out product.
Any chance you'll be doing a Summit presentation on Plot.ly?
**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ: http://lists.4d.com/faqnug.html
Archive: http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
**********************************************************************
**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ: http://lists.4d.com/faqnug.html
Archive: http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub: mailto:4D_Tech-***@lists.4D.com
****************
David Adams
2017-01-15 00:43:13 UTC
Permalink
As I said to Cannon about doing something on objects, I'll say to you about
Plot.ly:

Do it! Do it! Do it!

Maybe you and Stephen Orth could do a Plot.ly versus HighCharts
panel/smackdown. He did get Best in Show for his HighCharts presentation.
**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ: http://lists.4d.com/faqnug.html
Archive: http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub: mailto:4D_Tech-***@lists.4D.com
*********************************
Lee Hinde
2015-12-28 03:13:13 UTC
Permalink
assuming the only python use is your web server: pkill -9 python
Post by Kirk Brooks
At the moment I'm wondering how to write
the equivalent of pressing Control C in the terminal...
**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ: http://lists.4d.com/faqnug.html
Archive: http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub: mailto:4D_Tech-***@lists.4D.com
**************************************
Kirk Brooks
2015-12-28 20:40:43 UTC
Permalink
Thanks Lee. I was hoping it was something that easy.
Post by Lee Hinde
assuming the only python use is your web server: pkill -9 python
Post by Kirk Brooks
At the moment I'm wondering how to write
the equivalent of pressing Control C in the terminal...
**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ: http://lists.4d.com/faqnug.html
Archive: http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
**********************************************************************
--
Kirk Brooks
San Francisco, CA
=======================
**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ: http://lists.4d.com/faqnug.html
Archive: http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub: mailto:4D_Tech-***@lists.4D.com
***********************************
JPR
2017-01-16 16:52:49 UTC
Permalink
[JPR]
Post by David Adams
I managed to make it up
to the first day of 4D's very nice world tour last year and JPR showed a
renderer in 4D that had pixel-level responsiveness for millions of pixels.
I assumed he was using a technique similar-if-not-identical-to the second
one. (In which case, moving it to a WebArea + JS + Canvas would get you GPU
acceleration.) I couldn't come to the second day, so I never did find out
if that's how he was doing it!
Hi David,

The target was to display a map of a Data File, with a different color for each type of block. The Data File is built with 128-byte blocks. Because the size of some big datas, I had to show up to 50 million blocks, so 50 million pixels. The maximum number of block types is well below 256.

So I use a BMP picture, whose internal format is well described (https://en.wikipedia.org/wiki/BMP_file_format). I choose the simplest one (uncompressed, 8-bit pixels). The Pixmap is a BLOb, which has to be built in a particular interlaced way, for the BMP is not top-left to bottom-right.

The pixel-level responsiveness is done simply by getting the mouse position (click and/or move) and calculating which one of the 50 million possibles pixels is below the cursor. Then, knowing the pixel number, getting the info is a piece of cake.

And BTW, the limit of GET HELLUVA is not 1 billion, but 2 billions, for it's a longint...

My very best,

JPR
**********************************************************************
4D Internet Users Group (4D iNUG)
FAQ: http://lists.4d.com/faqnug.html
Archive: http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub: mailto:4D_Tech-***@lists.4D.com
*************************

Continue reading on narkive:
Loading...