Go has basically none of the mechanisms for sanely creating composable complexity that Scala has. It makes me really grossed out to think about something with the complexity that Finagle exposes as a Go library. Go is all about small, sturdy tools that work for most of the cases most of the time and that compose well with each other - IF - they are built into the standard library. And if it's not, have fun spending your time writing uni-typed interfaces for any kind of interop with desired complexity. Go is great for writing dead simple byte slingers, pipelines that will be rigid in the most dynamic possibility, guns that shoot JS and interact with services, etc. That's simply not Finagle.

My experience with Scala has been that an author's 'composable complexity' becomes a reader's 'wtf is this complexity'.

Go's philosophy has been to avoid complexity altogether in most of the tools they give you, forcing you to write lowest-common-denominator code that anyone can easily understand. Generics are sorely missed, but I really don't see additional expressive power from most scala features, just additional things I need to keep in my head for an occasional 5-line savings in the middle of some method (which IMO doesn't change the complexity equation at all). Would love to see some beautiful examples that change my mind.

Go's approach is to makes it easy to say certain things. Goroutines are great. But it also makes it difficult to say complicated things like List<T>. Good programmers suffer, but really it's just a few more keypresses, ala Java.

Scala makes it possible to say even more abstract things like T<int>. Not many programmers can really explain a higher kinded type, or why such a thing is useful. Novice programmers say silly things because they don't really know how to apply all of this abstract power.

Go is pretty wonderful because it lets programmers of varying ability work together effectively. Scala is teaching people new abstractions. Are they worth it? Maybe not in Scala. However, I think languages with powerful type systems will ultimately win, because you can lean hard on the compiler to guarantee correctness.

I also dispute the idea that only novices abuse complicated type systems. Actually I think it's the reverse... only people who know the language well can create something truly horrible. Don't believe me? Look at any of the C++ STL header files and experience Biij.

Of course, just because they can doesn't mean they will. But in a world dominated by people of average intelligence, working around the programming equivalent of a gas lamp with an open flame just isn't a good idea.

Scala has incredibly nice compiler messages and you can see Rust for an example of a language with complex types and short compile times.

These issues have _nothing_ to do with complicated type systems - you're projecting the failures of the C++ template system - failures which stem in fair part from the fact that it's not really a type system but simply a templater.

Finally, it's worth recognising that Scala goes to lengths to keep messy features inside their own compilation unit - for example, macros appear as functions to the outside and complex features such as implicit conversions must be explicitly imported for use.

I tried building our project in Go and Node.js explicitly to make it as easy as possible for people to contribute, irrespective of their background. It turned out that the additional complexity I had to pump into abstractions to make it happen made the code just as complex to understand.

It's quite possible that projects just have a natural complexity level and it's worth picking the language to match. There are certainly different times for different languages.

I'm quite sure the result will be quite different from what those kind of frameworks look like in other languages, but that could be interesting still.

One of the best examples of composability is the Reader and Writer interface. They're designed to operate on slices of bytes, so a package intended to read from a Reader can read from files, stdin, networks, sockets, and a myriad other systems with absolutely no changes to the package's code or its implementation.

    {"purchaseType": "Rent","price": 0.99,"title": "inception"}

    {-# LANGUAGE DeriveGeneric #-}
    {-# LANGUAGE OverloadedStrings #-}
    
    import Data.Aeson
    import GHC.Generics
    import qualified Data.ByteString.Lazy as BL
    
    data PurchaseType = Free | Rent | Buy | Subscriber deriving (Show, Generic)
    instance FromJSON PurchaseType
    instance ToJSON PurchaseType
    
    data Media = Media { title :: String, price :: Float, purchaseType :: PurchaseType} deriving (Show, Generic)
    
    instance FromJSON Media
    instance ToJSON Media
    
    jsonData = "{\"title\": \"inception\", \"price\": 0.99, \"purchaseType\": \"Rent\"}"
   
    main = print $ (decode jsonData :: Maybe Media)
    -- running main gets:
    -- λ> main
    -- Just (Media {title = "inception", price = 0.99, purchaseType = Rent})

    orderConfirmationMessage (Media mediaTitle _ Rent) = putStrLn ("Thanks for renting " ++ mediaTitle)

    Main.hs:19:1-100: Warning: …
        Pattern match(es) are non-exhaustive
        In an equation for ‘orderConfirmationMessage’:
            Patterns not matched:
                Media _ _ Free
                Media _ _ Buy
                Media _ _ Subscriber

Getting to the point, my real questions are:

1. What is Go's way of ensuring your function handles cases like this, or is there an idiomatic way of avoiding it?

2. How do you discern between MovieType's without resorting to just using Strings and compromising type safety?

    package main

    import (
        "encoding/json"
        "fmt"
    )

    type Media struct {
        Title        string
        Price        float64
        PurchaseType string
    }

    func main() {
        jsonData := []byte(`
            [
                {"purchaseType":"Rent","price":0.99,"title":"Inception"},
                {"purchaseType":"Free","price":0.00,"title":"Johnny Mnemonic"},
                {"purchaseType":"Buy","price":17.99,"title":"John Wick"}
            ]
        `)
        var media []Media
        err := json.Unmarshal(jsonData, &media)
        if err != nil {
            fmt.Println("error: ", err)
        }
        fmt.Printf("%+v\n", media)
    }

    $ go run main.go
    [{Title:Inception Price:0.99 PurchaseType:Rent}
    {Title:Johnny Mnemonic Price:0 PurchaseType:Free}
    {Title:John Wick Price:17.99 PurchaseType:Buy}]

    type PurchaseType byte
    
    const (
        Free PurchaseType = iota
        Rent
        Buy
    )
    
    func (t *PurchaseType) UnmarshalJSON(data []byte) error {
        var s string
        if err := json.Unmarshal(data, &s); err != nil {
            return fmt.Errorf("purchase type should be a string, got %s", data)
        }
        have, ok := map[string]PurchaseType{"Free": Free, "Rent": Rent, "Buy": Buy}[s]
        if !ok {
            return fmt.Errorf("invalid purchase type %q", s)
        }
        *t = have
        return nil
    }
    

    [{Inception 0.99 Rent} {Johnny Mnemonic 0 Free} {John Wick 17.99 Buy}]

tl;dr The problem for me personally is when you start making functions based on the type of media and sacrifice type safety if MediaType is a string.

That is just a really big one for me I wanted to share and get others opinion on. I guess I just feel like a lot of people don't realize how much having that stringly typed hole means. Perhaps others just disagree.

I would love to hear a counter argument for why others don't think this matters.

    switch (someEnum) {
      case foo:
        <something>
      case bar:
        <something>
      default:
        throw DeveloperScrewedUpException("Unknown enum val: " + someEnum)
    }

You may be confusing interfaces (which enforce type compatibility) and the empty interface{}, which is effectively a dynamic type in the Go world.

https://github.com/twitter/finagle/blob/master/finagle-core/...

https://github.com/twitter/finagle/blob/master/finagle-core/...

http://twitter.github.io/finagle/guide/Clients.html

A client is a layer of various components like load balancers and stats recorders, but they're also all configurable. Only using the expressiveness of scala could such a data structure be created to compose all of these pieces, yet still keep them modifiable and type-safe.

Writing in those other languages was indeed a hellish experience - it was very clear that they lacked the expressivity to model half of even just this very core piece.

This is something that keeps sending me back to Haskell, I'm glad to hear Scala applies as well.

I've been meaning to learn more Scala, is there any "Scala for Haskellers" literature you are aware of? I'm going to start searching for some, but you could know of one that doesn't come up in my search results.

Since I never spent time onboarding myself, I'm not sure I can be any help. :/

The docs website is amazingly good, though - the tutorial[2] covers most of the unusual-in-Scala bases very quickly.

[1]: http://www.scala-lang.org/docu/files/ScalaReference.pdf [2]: http://docs.scala-lang.org/tutorials/tour/tour-of-scala.html

We built a microservice architecture in Go at Hailo which runs approx 170 services. Abstracting core functionality into a 'framework' allowed developers to build and deploy services extremely quickly, and let the platform team get on with adding and improving core platform functionality, such as monitoring, instrumentation, tracing, provisioning, configuration etc etc.

One of the things I love about Go is that I don't have to learn all kinds of conceptually heavy frameworks to get real work done. I read the stdlib docs and get to coding.

I've also found that when I'm tempted to stray from Go idioms and do something too generically, I tend to get punished by the language. So I've learned to just suck it up and do everything the "Go way". I don't consider this a negative--rather that Go is just not very tolerant of other language styles. You have to accept this.

That's what I felt that overlapped with the GP's comments on composing complexity. It's not so much that you _can't_ implement multiple cases (not to the nth degree, just a handful), it's that the abstraction leaks painfully.

For me Go reminds me so much of Java 1. And it seems to be getting pulled by the community in the enterprise Java direction e.g. generics, annotations, dependency injection, broad frameworks and the simple, tightly focused tooling direction.

* Go had annotations since pretty much day 1. Look at the annotations for JSON serialization, for example.

* Generics is something that Go is probably not getting. It's just become a troll topic at this point since everything that could be said about this topic, has been said.

* Dependency injection is something that Go isn't getting. It's a Java solution for a Java problem. Create well-thought out code with clean interfaces and unit tests... problem solved.

* Broad frameworks: it's just not going to happen. As a lot of other commenters have said, Go just doesn't support the level of abstraction that you need to create a FactoryPatternPatternStrategy. And that is a good thing, in my opinion.

* Tooling: I have no idea what you mean really. Java's tooling is hardly "simple" or "tightly focused" (have you ever USED Maven?) and Go's community didn't "pull" the golang authors to create simple tools... they've been simple and focused since day one.

Request => Future[Response]

I don't know what the equivalent would be in Go but I can't imagine its any clearer than this.

    type Service interface {
        Handle(req Request) Response
    }

Then make Future generic so that you can use all the existing combinators out there, like `sequence` which turns `List[Future[Response]]` into a Future[List[Response]]`.

Careful implying that using futures and promises are incorrect unless you can prove that channels can do anything futures and promises can do while doing it better.

Do you have any examples?

http://golang.org/doc/effective_go.html#channels

And I don't think the problem domain is intractably complex; I think we have enough collective operational experience to have informed opinions, and I think Go's type system may be sufficiently sophisticated to encode those opinions.

I gave a talk at the (unaffiliated) Hacker News London Meetup on composing a microservice architecture and why I think a framework to bind today's disparate components needs to exist: http://vimeo.com/118895501

I don't so much believe that any language is perfectly poised to implement this, however - we're working on a framework[1] that focuses on making every language a first-class participant.

Furthermore, it's self-hosting, with most of the core itself implemented as communicating microservices.

It's possible to reimplement bits of the core in different languages!

We've been waiting to finish a polished implementation before releasing anything (and talking to individual companies about their needs), but I'm starting to think that's a bad idea. :)

[1]: http://wym.io/

Also,since languages are vastly different(in features), don't you think it will limit what your framework can do?

Great talk anyway.

I believe however,that having a central authority that orchestrates services is better than micro services that can talk directly to each other.In my opinion,some kind of mediator is mandatory,one that registers services and manage service communication.

- What's the difference from a spec?

There used to be a FAQ on our site emphasising that at its core _wym_ is a spec. Our reference implementation, itself made up of multiple pieces (wymd, wymsy, oneweb, ...) provides a way to actually make use of that spec and to validate all the guarantees of the strict interfaces.

Also, I'm not a fan of entities who would define a spec in isolation without a reference implementation alongside it - where's the proof that you're making the right decisions? We provide that implementation to keep ourselves grounded.

- Won't multi-language limit features?

Not really.

For one thing, since we're self-hosting we can't afford to lack power and expressivity. For example everyone has the ability to refer to types (as per generics) else we wouldn't be able to implement the core. ;)

Our interface language - it's quite simple, to make the addition of new language bindings trivial, but we expose built-in services to use more powerful features simply by switching which nodes you're holding on to - there's a data-implementation divide, where a node needn't understand a feature in order to deal in its data.

The net effect is that language bindings can be created trivially and evolved to support features as a good way of dealing with them is found (e.g. in the simplest case, Javascript Int64 support).

- Having a central authority is better.

This is an interesting one :D - I agree, in a way. A wymd cluster self-starts a consensus using a lock service, performs master election and then uses this consensus service for registration and discovery.

We have an elected distributed authority. :P

> function (){ return React.renderComponentToString(Application(null)); }

Just looks a bit messy tbh. ;)

I'll look into it. Thanks. :)

If you figure it out, please let us know/write it up -- being able to use react and have it work without javascript would be a very nice feature.

I'll upload our build setup right about when I write a blog response to the OP's post. :)

The chief point to note is that the components need to render as you want non-JS viewers to see them when they're passed the default props/state.

And no, I'd rather not 'vendor' source-snapshots of a logging package into every project I ever work on.

What I mean is... there's a clear and easy way to version libraries in Go - you encode the version in the import path in some way. When the version changes, the import path changes (which means it's a completely different dependency, as far as Go is concerned). How you get those versions in the import path is up to you - you can use gopkg.in, you can host your own redirector, or just have different repos for each version. It's really not rocket science.

For example, I released github.com/natefinch/lumberjack as v1... when I decided I needed to break the API, I released v2 as gopkg.in/natefinch/lumberjack.v2 (which redirects to the v2 branch of the same repo). The very fact that I'm using version numbers indicates I have a commitment to keeping my library stable for third parties. Now, you don't have to trust me, you can use godep or another tool to pin your code to a specific revision of my repo, and if you're really paranoid, you can use godep to vendor my code into your own repo, to insulate you from the possibility that I delete my repo in a fit of anger. (though in theory, that would only stop you for a short period of time, since you should have a copy of my code on every machine that has ever built your code that uses my library).

I just don't understand why everyone thinks versioning in Go is a problem. It's not. It's solved... seven ways to Sunday. It's just solved in a slightly different way than other languages solve it... but so what?

I think the thing that conflicts people most often is that there is truly no "official" opinion on the subject aside from what you mentioned: use a different import path.

There is no official stance on package versions, because the core Go development doesn't really need/use it.

The Go 1 compatibility guarantees no backwards incompat changes to stdlib. Experimental package development occurs at golang.org/x/foo ("x" for experimental, presumably). Those packages are unstable, and there are no versions. If they change at all you have to update.

People follow the 'official' approach blindly, but aren't experienced enough to know what the tradeoff's in the official approach are.

As I said, there are tools that let you freeze the exact commit number your code depends on, which effectively gives you frozen minor version numbers if you want it. And of course you can also copy the code into your repo with godep, and not need any external services outside your own VCS.

Eh ... why? In some languages, it's standard to run source/binary compatibility checks against new versions and fail a build if the new version doesn't satisfy the requirements.

"I suppose"? This has been done for years already. There are even GUIs for that (so that people don't have to use the CLI) which tell you "between 1.2.3 and 1.2.4 you did the following: ... This will break compatibility in the following cases: ..."

So it's better to not even try?

Additionally, for many changes it is reasonable and possible to check whether behavior has changed. Not every function has an unlimited number of inputs and outputs.

That's one of the biggest reasons I avoid Go and PHP: It's not only that the developers ignore existing solutions if they haven't "invented" them themselves, but that they don't even care about improving things where no "perfect" solution is available.

Ignorance and arrogance is a combination which doesn't make me trust them, and products based on their software. (OwnCloud, Docker, ...)

The problem I have is when other developers say "How can anyone possibly live without this?!" People have been living without X or Y feature some other language / toolchain has for a long time, and doing pretty ok. That doesn't mean we don't want that feature, just that we don't think it's the end of the world if we don't have it. Everyone seems to have their own pet "I can't live without this" thing. The thing is, when you've been coding for a long time, you start to realize that you can live without pretty much any feature. It may make some parts of your job harder, but unless we're talking about dropping down to assembly, I can deal with pretty much any language.

Go developers seem to be emphatically content with not using the best tool for the job. That makes sense, otherwise those developers wouldn't have ended up with Go in the first place.

For me it's not a question about "I can live without X?", but "why would I want to live without X?". I want the best tools to solve a job. Not some of the best tools. _All_ of them.

They're seeing something that you are not seeing, most likely an appreciation of simplicity.

You obviously don't read the Golang mailing list ... :-)

With that being said, I'd recommend http://gopkg.in, or if you prefer hosting yourself https://azul3d.org/semver

Disclaimer: I created `azul3d/semver`.

This is basically mandatory in most enterprises since often you can't just go into the wild to pickup new dependencies. Plus keeping build artefacts managed is pretty important.

It is fine to assume the availability of source code when you're working on tiny web microservices... and you're only depending on things like Gorilla, or whatever the leading Sinatra-clone is this week. However, a typical "enterprise" project has numerous proprietary dependencies for which the vendor doesn't want to ship source code.

It will be interesting to see whether the Go community eventually adapts to fit enterprise culture, whether enterprise culture adapts to fit Go's model, or whether the two worlds never really come together at all.

What go has is Godep (https://github.com/tools/godep) which with a single command can copy all your project's dependencies into your repo. from there on out, the dependencies are just part of source control like anything else, so you don't have to open up your build machine to the external internet, for example.

It also has a fine grained security model and allows for security evaluations of libraries (check for outdated/insecure etc).

Sorry didn't mean to rant... Love Go its the future, use it.

I used to think and hope Go was the future, and I agree it's better than node.js's callback hell, but I no longer hope Go is the future.

Something like Nim[0] is an improvement in every way.

Not my main point, but I personally moved from Go to Haskell. So, If you don't mind educating your developers Haskell, Scala, or Ocaml offers you true type safety and compile time correctness.

0: http://nim-lang.org/

We are extremely open to suggestions/feedback.

One key point I'd like to point out though, is that OP specifically mentioned the go kit being targeted towards a company with "100–1000+ engineers". I think this is highly on point - if you have less engineers than this you probably don't have enough resources to maintain these sorts of systems yourselves and this is where we think we can help out.

Ouch! I work for HP and while I agree that we're not exactly a nimble, fast moving startup I think it's not a good comparison. IBM and HP still build a lot of physical things while the listed "modern" enterprises are primarily SW. Google would be the exception here but from an engineering point of view Google uses it's vast war chest to play around with very cool HW but isn't much concerned with running a business based on that. So, I hope Go has a good run of it but it's not going to build your next printer.

A lot of these things aren't golang-specific, though. Ideally it would be easy to swap in services written in different languages, and the conventions stay the same.

However, not only did we need something like this 5 years ago, we needed it to be language/platform agnostic then. We need it more so now. We won't be able to function without it 5 years from now.

The reality is Pandora's box has been opened there is no going back. We have software we need to run in production at scale that is written in a multitude of languages. We have our Python/Perl monitoring systems talking to our Java search APIs talking to our NodeJS frontend APIs talking to our legacy C#/Java/C++ applications over messaging systems written in Erlang/Ruby/Java/Go.

Every single one of these platforms need this functionality and everything needs to run in a cutthroat environment with fewer people, resources, and money to manage it.

We need performance, discoverability, security, scalability, deployability, testability, reliability and most importantly EASY. Not enterprisey over-engineered bullshit. I mean real easy to use plug and play, simple APIs, minimal JSON/YAML, small focused tools simple.

We need to do it in a way that is portable to every cloud provider so we can get the best bang for our buck and keep the cloud providers honest and avoid lock in.

We need to do it in a way that makes the "hardware" vanish. We don't want to manage hardware anymore. We want to deploy our software and we want to know it will work and we don't want to compromise functionality (ala Google App Engine or being forced into the "Tomcat 7" box).

I applaud this line of thinking, but I don't think it goes far enough. In 5 years time, I don't think we'll deploy servers anymore. We'll deploy services and those services will be in whatever language and those services will find each other and communicate with each other over the network and will be reliable, secure, and scalable.

Most importantly, it needs to be open. Truly open. We don't want to trade one master for another.

That's the software I would like to see. This could be a foundational component of something like this, but it's only the beginning.

I know about CoreOS/Docker/Mesos/Marathon, etc. Tools like those are an important piece of this puzzle, but not the only piece. They are a vast improvement over past initiatives (cough openstack cough), but they too are only a step in the right direction. We still have a lot of work to do to realize this vision of the future.

It was also an overly complex piece of shit.

People learn nothing from the past.

Not least I talk about how it's important to build something that modularises complexity and above all else, focuses first on being _usable by humans_ rather than on technical ascension.

_Some_ of us _are_ listening.

[1]: https://vimeo.com/118895501

[2]: http://wym.io/

Hopefully they'll be too busy writing AbstractFactoryBuilderServiceDispatcherProxies to actually do any damage to production systems...

What he proposes here looks more to me like a standard library for building distributed systems, which makes a lot of sense to me.

I was thinking of using heka for a logstash-like project and just wanted to see if you had found several know issues, or that monitor really shined, or that maybe the feature set you needed just didn't overlap to the point they were comparable for your needs?

Not necessarily bad but.

In fact this is similar to how Go replaces the OS scheduling and message passing.

Maybe its a necessary path to evolution. Til the OS itself can get fixed without "it doesnt run X or Z" problems.

Somehow we will end up losing the features of the language that make is currently so endearing.

    package main

    import (
        "fmt"
        "net/http"
    )

    func main() {
        for _, url := range []string{
            "http://httpbin.org/status/200",
            "http://httpbin.org/status/200",
            "http://httpbin.org/status/200",
            "http://httpbin.org/status/404",
        } {
            fmt.Printf("%s: %d\n", url, checkUrl(url))
        }
    }

    func checkUrl(url string) int {
        response, err := http.Get(url)
        if err != nil {
            panic(err)
        }
        defer response.Body.Close()
        return response.StatusCode
    }

What if we don't care about the errors and just want to return nil or the value. The caveat is that when any of your http requests get a non 200 response code no further processing is done and nil is returned.

    if !checkUrl(url){
        fmt.Printf("Non-200 response for %s", url)
        break
    }

Go has another advantage over Scala (and any other JVM langauge), which also happens to be it's killer feature - green threads, the ability to write non-blocking concurrent code in a synchronous style without callbacks. Yes, I am sure Scala has some "leaky abstraction" over what are essentially just callbacks, as does Java, but it's not quite the same.

"Green threads", per se, aren't an advantage. M:N threads are potentially an advantage, since they allow both hardware parallelism and concurrency with a greater degree than the degree of hardware parallelism without the overhead of native threads for each concurrent task.

OTOH, nothing stops either JVM implementations (when compared to the underlying platform) or JVM languages (when compared to the JVM itself) from using an M:N threading model.

> Yes, I am sure Scala has some "leaky abstraction" over what are essentially just callbacks, as does Java, but it's not quite the same.

Green threads (whether 1:N or M:N) are a (potentially leaky) abstraction over what are essentially callbacks, too (while a well-designed green thread implementation can avoid this for code written entirely in the language, for calls linked code written in other languages that isn't specifically aware of and designed to cooperate with the green thread implementation, there is the risk of actions which block the underlying native thread, which will then block all green threads that are running on the same native thread.)

It's not a one-size-fits-all type of game here. So having one choice will essentially lock you into what you can do with it.

I also wouldn't say a Future, which is a Monad, a leaky abstraction. You don't really need to understand monads to use them, in fact, a lot of people use what are monads all the time, without knowing it.

I thought this was software transactional memory? Java has an implementation?

There isn't any requirement for red threads in certified JVMs and a few of them do offer green threads, which is actually how the first versions used to do threading.

However there exists Quasar for the JVM which seems to be able to emulate something similar to green threads with the help of bytecode weaving (and inserting preemption points).

There is much more out there than OpenJDK and Oracle commercial JVM.

If you want a real product example,

http://www.is2t.com/products/specifications/

You have different problems and pressures when you don't have hundreds/thousands of highly experienced engineers.

Google, Amazon, Twitter, Netflix, and SoundCloud are for engineeers living in Silicon Valey.

How is the sector at all relevant? (Software is eating the world, and all that.)

Google, Amazon, Twitter, Netflix, and SoundCloud and a few others are pointed out in HN as enterprise examples, and they are.

However outside the HN bubble, which I tend to associate with SV, the enterprise looks a bit different.

It is those Fortune 1000 companies, most of which don't have software as main goal, are happy to outsource anywhere in the world, still run mainframes and commercial UNIX systems, have rigid processes in place and so on.

This is the enterprise I know from 20 years in the business. Amazon et all might be the modern enterprise, but they are a very tiny piece of the pie for anyone working on IT around the globe.

It's not about recreating everything in Go "just because", it's about being a Go developer and wanting those tools to be available to you.

Other threads in this discussion bring up Finagle's shortcomings, partly due to certain features not being addressed or included by Finagle's authors due to them being low priority. In that case what you want may not already exist, but may work as inspiration for another incarnation.

I've personally run into many situations in which porting a lib to Go is easier -- and more efficient -- than my brain's context switching between multiple languages in order to fix issues with the original lib.