> image is not recreated from scratch every time and it is more than just a cache
Yes, some vm & image & sources & changes can be taken as the base implementation for development purposes -- a persistent cache.
The state of whatever IDE tools were in use will be saved -- is that what makes you say "more than just a cache"? If I sleep a windows desktop is that more than just a cache?
> changes first happen in an image
What if I write a plain-text source code file using Notepad, and use Smalltalk file handling and byte code compilation and command-line argument handling (packaged in the image) to write the result of a computation to stdout (and quit the image without saving)?
If I sleep a windows desktop is that more than just a cache?
yes, so basically what i meant here is that a cache just stores data, but it doesn't store the whole application.
this is significant in that i can shut down an application (say my webbrowser), then i can upgrade it to a new version, restart and and the application will reinitialize itself and load data from the cache, but now i have a new version of the application.
whereas if i put my laptop to sleep, or better yet, hibernate, then the whole state of the laptop is frozen in place, and i can't do anything to it until i run it again. same is true for smalltalk images.
What if I write a plain-text source code file using Notepad, and use Smalltalk file handling and byte code compilation and command-line argument handling (packaged in the image) to write the result of a computation to stdout (and quit the image without saving)?
you could be doing that, but then you would use the image as your IDE and runtime environment, but not building the actual application in your image. so you wouldn't using what i have been taught is the traditional way of doing smalltalk development.
i am not trying to be pedantic here. it does not matter either way. i just find the smalltalk image approach interesting because it forces you to think about software development in a different way.
this mattes to me because i am working with a web development platform (written in pike) that uses a similar approach. albeit more by accident than intentional. the developers of the platform added support for programmable objects that are stored in the platforms database. these objects can change the behavior of the platform itself, like plugins, but because they are stored in the database they can be changed at runtime, like a smalltalk image. and all the same implications for doing that apply here too. the database becomes more than a cache. and in theory the whole platform could be rewritten such that almost all of its code is stored in the database and only a small bootstrapping system needs to remain outside. this is simply made possible because pike can load and update code at runtime and code changes can be applied without restarting, just like smalltalk.
the downside of the image approach is that it makes upgrading the base image harder, because there is no clear distinction between the base image and any user added changes. i kind of have to take extra steps to pick out my changes and apply them to a new image.
it would be interesting if that process could be improved. it probably would require some compartmentalization just like an OS where i have the base OS, my home directory and the system configuration. i can take a disk image, upgrade the OS and the rest still works. it would be nice if upgrading pharo for example would work the same way.
btw: thanks for the email. i have to ask, how did you manage to reply to a comment more than a month old. normally the reply function is disabled on comments that are 14 days old.
Is that in-conflict with a reproducible build process or can we have both
in part the question is what makes a build reproducible. what actually needs to be reproduced? the point of a reproducible build is that a version of source code always produces the same binary.
how do you make reproducible builds in smalltalk? reproducible builds depend on what goes into the build process. so they depend on the compiler and build tools. in smalltalk those are all in the image, and the question is then what happens when i load code into an image. am i getting that right? i am not so familiar with the details here, but i would guess that it depends on how smalltalk compiles the code and how the import process deals with timestamps and source paths, etc.
however if i work on my source code within the image and i share the code by making a copy of the image then the image is the source and the binary and there is nothing to reproduce. your copy and my copy of the image are going to be identical until one of us makes changes to the image.
i'd be curious to learn more here. outside the smalltalk world my editing tools do not affect the reproducibility of the builds of my code. in smalltalk. just getting a new version of the code browser would change the build, wouldn't it? how do you track that or keep that separate?
Port "user added changes" from the source code archive to each vendor release
right, but that's the "wrong" way around from the perspective of a desktop. i don't need to port my code to new versions of VS Code or vim or which ever tools i use to develop. only smalltalk forces me to do that. so i don't mean distinction in the file structure but distinction in the code dependency.
(i would never have considered to ask for being allowed to make a late reply, especially in this case. it is unlikely that anyone else is going to see our conversation. we could have just continued over email. but hopefully we can dig out some worthwhile details that not only me but anyone searching can learn from)
it sounds like you are asking a question, but there is no "?" at the end, so i am slightly confused. if it is a question, my answer is no, because the tools i use to read or edit the code are completely distinct from the build tools.
what affects the reproducibility is the compiler, and that's the issue with smalltalk. you can't upgrade the IDE without upgrading the compiler. say if i use pharo, and switch to a new version of pharo then i get a new version of smalltalk, and i can't compile my own code in the image with the old version any more.
based on that i don't understand how you even enable or test reproducibility in smalltalk. i'd like to learn more about that.
(re: vanity. since a while i have wondered how i would go about creating the longest running discussion thread on hackernews. and how long i would be able to keep it going. i think we are off to a good start here. just remember to peek in here once a while to see if there is a response. and if there isn't feel free to poke me by email. (unless we decide that there is nothing to add))
> … can't upgrade the IDE without upgrading the compiler.
When packaged together IDE & compiler can't be upgraded separately (without doing the work to in-effect make separate packages).
> … can't compile my own code in the image with the old version
So we could try to compile own-code' in image' with compiler', and we could try to compile own-code' in image" with compiler", but we want to try to compile own-code' in image" with compiler' ?
Not without doing the work to in-effect make separate packages. (Say we could port compiler' to image" but would that mean trying to compile compiler' with compiler".)
And now we're back to what does "traditional way of doing smalltalk development" mean because supposedly "Team/V could forward and backwards migrate versions of Smalltalk “modules” within a running virtual image."
> the point of a reproducible build is that a version of source code always produces the same binary.
Scope: does "a version of source code" just mean own-code or does it mean sources+changes.
i have just been reading https://news.ycombinator.com/item?id=48081245 about reproducible builds in debian, and that reminded me of this discussion. truth is i still struggle to understand how reproducible builds with smalltalk images even work.
in debian, a build is reproducible if the checksum of the resulting package of my build matches that of your build. how do you do that in a smalltalk image? you take the checksum of what? which objects or rather entities do you compare, and how do you compare them, to verify that a build is reproducible? it can't be the whole image, because that is guaranteed to be different somewhere, and thus the image checksums won't match. so how does that really work?
there is another thought that i just realized. what are reproducible builds with runtime compiled languages like python, ruby, perl, even javascript? the point of a reproducible build is to verify that the binary i use, or that the package i use is based on the same source that i have. python et al don't have binaries. so my reproducible build check is a diff of the source tree. if in smalltalk the image is my source then i care about knowing that the image has not been modified. that's impossible because the image changes just by using it, unless i never save the changes. what's left is exporting the source and running a diff on that maybe.
> … the point of a reproducible build is to verify that the binary i use, or that the package i use is based on the same source that i have.
I have been talking about how to reproduce a particular build, not how to verify.
From the same initial state, perform the same sequence of actions, implies arrive at the same final state. Reproducible.
"Retaining your old change logs gives you a record of all the changes you have made to the system. This will prove invaluable when you receive a new release of Smalltalk/V.
… In short, back up the image and change log together, and you shouldn't have any problems."
page 285 Smalltalk/V 286 Tutorial and Programming Handbook 1988
ok, it seems we are talking about different things. "reproducible builds" generally refers to verification. so i misunderstood what you meant.
so we actually have to go back to this question:
> traditional way of doing smalltalk development
Is that in-conflict with a reproducible build process or can we have both?
to summarize, i was talking about traditional meaning to make changes in an image and then exporting them if needed. and then i was wondering if local changes and the base image can be kept separate, and by that i meant "within the image" so that, when you look at an image, or when you use it, browse the code, you can easily see what are changes and what is the base image. kind of like in an operating system where i can clearly distinguish system files from user files. so effectively that would suggest some kind of ownership markers, or, alternatively, it needs versioning of objects within the image.
i am not suggesting that smalltalk needs to be changed, but i am trying to work out the limitations that affect the workflow. essentially, because this distinction is missing, there is no way to reproduce a build (in your sense of the meaning) without exporting the code and re-importing it in a new image. actually, object versioning within the image would be a really useful addition to smalltalk. it would allow you to easily go back to old versions. of course it would also cause the image to grow, so there would have to be some efficient way to store that. for one, old versions don't need to be kept in memory so they could be in a secondary storage. or put differently, smalltalk could keep track of the changes in the changes file as it does, but in such a way that i can browse and load in old changes from within the image. i don't know if that is possible now.
the image based server system that i talked about earlier btw does have an ownership system and object based versioning. that reveals some interesting potential.
btw this lack of distinction between base image and changes in the image, especially the lack of ownership information i believe is one reason why linux distributions didn't pick up smalltalk as soon as it became available under a free license, because the license could not actually be verified across all objects. especially older objects that had the source purged to save space.
thanks. it's going to take a while to find time to read those. likely longer than the two weeks available to reply here. let me just say that my experience with smalltalk is limited to playing with squeak and doing some coding with pharo. (i made some video workshops life coding through some tasks, if you are curious).
i am also not suggesting that smalltalk can't be changed, just that i didn't want to assume that my suggestion is the right way, and has to be done, or else. more like: how about this? is that feasible?
cuis is interesting, there are other branches that i find interesting too, such as strongtalk. it's a pity that they receive even less attention than smalltalk itself.
i wish i had more opportunity to work with smalltalk, but nobody is hiring ;-)
i didn't see that comment, sorry. my primary purpose here is to learn more about smalltalk and the image based development model. you brought up a few interesting things that i have questions about. reproducible builds is one of them. and by question i don't mean to verify the validity of claims but to ask about them to learn more.
> image is not recreated from scratch every time and it is more than just a cache
Yes, some vm & image & sources & changes can be taken as the base implementation for development purposes -- a persistent cache.
The state of whatever IDE tools were in use will be saved -- is that what makes you say "more than just a cache"? If I sleep a windows desktop is that more than just a cache?
> changes first happen in an image
What if I write a plain-text source code file using Notepad, and use Smalltalk file handling and byte code compilation and command-line argument handling (packaged in the image) to write the result of a computation to stdout (and quit the image without saving)?