Write a crawler that examines all local links, and builds a graph (it needn't be anything fancy; a collection of nodes and edges is enough, but you'll have to detect cycles when building the graph), and then print it out in graphviz .dot language.
Then you can feed it into neato [either through a command-line pipe, or construct one using Python's subprocess module], et voilà, instant pagemap! :)
You can also use some of the rendering modes and scrape position data for the rendered nodes, if you want to generate a clickable imagemap.
I tried rendering this with "dot -Tcmapx -oob.map -Tgif -oob.gif" but dot segfaulted after 70 minutes. The code as posted at gist only outputs nodes for articles written by Eliezer in an attempt to make the dot file a manageable size. Tips/fixes appreciated.
Yeah, it's that thing with the segfault that I worried about. "Highly interconnected." Thanks for the graph, which I may be able to use for my own purposes even if I can't show it.
There's got to be standard graphing tools for things that are highly connected...
When Daniel J. Bernstein made his DAGs of standard crypto algorithms (http://cr.yp.to/cipherdag/cipherdag-20070630.pdf) he mentioned that they'd crashed every standard drawing tool he tried. He makes a reference to "...my own drawing tools, which are much more careful in their use of memory." Unfortunately, I'm not sure where to obtain DJB's drawing tools.
Graphviz only renders a graph which is represented as per dot format. There are other programs: dot, neato etc which can produce jpeg,png,gif... formats through command line tools.
I guess what he meant was tools which can do this for a particular site.
I can't think of any but as a previous poster said - a crawl of desired depth for all local pages, then feed the access log to "statviz" to generate a neat dot file along with the links.
I am home for winter break and would be willing to write this for you in the name of science. Plus, it seems that you know Marcello Herreshoff, with whom I took Computer Science in high school. Email me the requirements for your tool to <zackster>:at:<gmail.com> please.
A tool like this would be ideal in locating the moment when a person stops producing original thought, and starts repacking his old thoughts in new structures.
One or two links to an old post would not deteriorate the quality or freshness of content. In fact a lot of times one might be interested to get broader idea of the author stream of thoughts, for the ones not interested thye just have to blind to hyperlinks.
May be it would become restructured content if there are too many outward references like the... "Undergraduate" post. But even that when you look into this post the hyperlinks are for keywords like "essays", "hacker", "computer science", what you "love", hack a "blub" on windows ..
This is cool - led me to http://www.paulgraham.com/javacover.html in which I read "Historically, languages designed for other people to use have been bad" which makes me feel a little less insane for spending the morning working on my "pet language".
It measures some combination of how general an essay is, how early it was written (I usually add these links when I first write something, so most links are only backward in time), how much I like it, and sheer randomness (at least half the time I don't bother to add crosslinks).
Larger numbers of outgoing links probably don't imply recycling. If anything it's a sign I think an essay is good and will get lots of readers, so it's worth trying to spread that traffic around. (These predictions are often wrong, though. I find it practically impossible to guess which essays will get lots of traffic.)
Is it not worth considering putting "forward" links into
older essays, so that when you write an essay it gets
referenced by other essays, even when they're older?
Perhaps you don't have time, but would you consider
adding links that other people suggest?
It would be interesting to do to this what I do with my
SiteMap, and colour nodes darker if they have more
traffic.
I like the idea, but it would be more meaningful if you could link for semantic reasons - that would be a very nice tool, could even be used by authors who want to put a book together from various essays they've written, etc. Or help you focus on an area of writing you're interested in by the author. I mention this because looking at the graph you can see that things that are related are not necessarily linked.
This would be really cool. Can you suggest a way, given
two essays, to decide if, semantically, the first should
point to the second? If an author could use a tool like
that it would mean that links could be discovered rather
than inserted by hand.
That would be valuable ... you've made me think ... I may
have a way of doing something close enough.
Gregor Heinrich has a good paper on Latent Dirichlet Allocation, which I believe is an extension of Latent Semantic Analysis. It is a model which can be used to group documents based on semantic content. He gives the mathematical details and the "punchlines" for implementation. The model takes as input a collection of documents and outputs a topic label for each word in each document. The documents can be plotted in K-dimensional space, where K is the number of possible topics, by using the proportion of each topic in a document as its coordinates. Documents which are closer to each other have topics more similar to each other. You could then use your favorite clustering algorithm or a simple distance threshold to decide which should documents should link to each other.
In a way, there is a gross semantic linking in a Q/A type of way with the titles, e.g., Why do Startups Condense in America --> The Power of the Marginal. What Can Business Learn from Open Source -->How To Start A Startup + Inequality and Risk.
I wonder if someone can use the PageRank algorithm or something similar to determine which essays are the most prominent. It's hard to tell with this graph which is the most connected essay - is it "Taste for Makers", "What You Can't Say", "Great Hackers", or something else? It would also be interesting to correlate this with the Google PageRank for these pages, and traffic data (if it's available).
Here are the ones that get the most traffic, with daily page views:
1760 Stuff
770 What You'll Wish You'd Known
403 How to Start a Startup
365 Why to Start a Startup in a Bad Economy
349 Why Nerds are Unpopular
294 How to Do what You Love
274 Web 2.0
247 Great Hackers
207 How to Make Wealth
187 Lies We Tell Kids
"Stuff" is my sending the given link to my female relatives to dissuade them from, or to reduce their practice of, being packrats. I'm pretty sure there are other users out there that do the same...
I've just pulled the description from WikiPedia, implemented pretty much that, and that's what I did.
In essence, each page gets set to 1.0. Then distribute 80% of that down each outgoing link, and 20% to everyone equally. Lather, rinse, repeat.
I think that's pretty much the algorithm. Pages pointed to by pages with lots of juice get lots of juice. Pages pointed to by no one, or only pages with little juice, get little juice. It's linear algebra thought of as network flow.
Yes, but doing what you've done here gives the entire site,
whereas implementing it from the definition allows you to
compute the actual values, as I did.
Of course, Google have twoke their algorithm so it's no
longer exactly as they originally published, and they're
not telling anyone what it actually does. Using the
"clean" version is at least transparent.
