Why I Switched from Python to Clojure (2016)

I spent some more time with Python and started to realize that I needed something more. Here's a brief list of the reasons why I switched to Clojure.
Article word count: 1081

HN Discussion: https://news.ycombinator.com/item?id=19477990
Posted by bsg75 (karma: 4902)
Post stats: Points: 123 - Comments: 86 - 2019-03-24T18:43:17Z

\#HackerNews #2016 #clojure #from #python #switched #why

---

Article content:

First – a bit of background. When I first started learning programming, I started with a course in high school that focused on Java. From there, I decided I wanted to work for a start-up and I had to learn python (I was naive). I started playing with Python and it just felt right – at first. I spent some more time with it and started to realize that I needed something more. Here’s a brief list of the reasons why I switched.
Interpreted Language
Python is a dynamic language. There’s a massive list of Pros and Cons for this, but I’ll highlight a few of my concerns.
\* It’s easy to introduce bugs on uncommon code paths that would be much more obvious in a compiled language.
\* The performance of an interpreted language is almost always slower than a compiled language.
Those things aside, I LOVED being able to quickly run code with a fast turn around time. When testing, I didn’t have to worry about compiling anything and, when I was developing with Flask, tools like Live-Reload were things I didn’t want to give up.

With Clojure, I found that I could have the benefits of an Interpreted Language and the benefits of a compiled language, too! By leveraging Clojure’s REPL, I’m able to quickly test code before/while/after I write it. You’re able to switch into your Clojure codes namespace and test any functions you’ve written, as well as get the source code for any specific function, or check the docs for any function too! If you’re not using the REPL when you’re writing Clojure, you really should give it a try! If you’re using Leinengen, you can run lein repl from your project directory!

With Clojure, you compile down to Java Byte Code when you run your application or library. You get the benefits and performance optimizations of a compiled language, but can treat it like an interpreted language when developing. It’s great!
Classes Vs Data
When I moved from Java to Python, I fell in love with Mutable Dictionaries. I became very intimate with the dictionary API and defaulted to using them instead of classes because they often met my needs on their own and honestly, the “implied self parameter” on python methods turned me off. Dictionaries are basically just a map data structure.

When I learned about Clojure, I was baffled by their lack of classes, but felt liberated by the thought of only writing functions in namespaces. I decided to give it a try and immediately started thinking of Dictionaries in Python and how I didn’t enjoy writing classes. With Clojure, I’m able to focus on what the data looks like, and not on how an object is expected to act (What’s an instance of a class but a hash-map with some namespaced functions anyways?).
Immutable Data & Stateless Code
Like most now-functional programmers, I had THE incident. I had a class with several methods, but only two were particularly important. Both methods were getters, but the 1st method updated the value that was returned by the 2nd. This was extremely misleading when the code executed. In fact, almost all of the bugs with this code revolved around this theme. I didn’t know what to do and I thought this was the expected way to write code. I later read a great post on immutable data and tried writing my python code this way. This worked great – until I started working with other developers.

With Clojure, you have to fight to use mutable data structures (that’s not to say it’s impossible, it’s just not the default). More importantly, if you’re working with Clojure developers, chances are they’re very familiar with the benefits of immutable data. I say “fight” because you do have to put forth some effort to do it, but you often don’t really want to.
Community
Don’t get me wrong, the Python community is great. Probably one of the best ones out there, actually. By comparison, Clojure’s community is very small. As you know, the communities are the groups of people responsible for building libraries for the language to use and helping people who get stuck with strange error messages or confusing domain problems. In terms of libraries, Clojure has extremely nice Java Interoperability (Clojure can call Java code directly from itself). This means that Clojure has access to ALL Java libraries and any clojure libraries as well. That’s a vast amount of libraries!

The support and documentation for most Clojure things pales in comparison to python, it’s true, but what I have noticed is that it matters where it counts. Clojure’s core API is relatively lean and extremely well documented. Additionally, the Clojure community didn’t experience the weird split that Pythonistas had with Python2 and Python3 (I’d imagine this is worked out by now, but I haven’t kept up with Python since I stopped writing it for anything more than scripts).
No __init__.py or __setup__.py
A shitty reason to dislike python, but these both irritated me a LOT.
For those who don’t know, and init.py file is used to declare a directory as a python module, while a setup.py is used to define how other scripts should interact with yours. That being said, neither of these have to do just that. In fact, I feel like the expectations for what each should do is extremely vague and troublesome. Some people prefer to import modules into their init.py or even add convenience functions there, too. It’s often easy to overlook these files, but if you can’t find the definition for a particular function or module, check the init.py

As for setup.py, you should probably just call the setup function from distutils.core. But, you can do more in that file – so much more! In fact, the actual documentation on this is pretty vague on what all should exist in a setup file, and can be found [1]here.

In Clojure, everything is handled in a nice project.clj file, if you’re using Leinengen. That being said, you can also use a nice, structured .gradle file, if that’s more your cup of tea. A project.clj file is defined similar to a package.json if you’ve ever used NPM for a JavaScript Project.
The Switch
To sum everything up, I switched predominantly because I wanted to speed and comfort of the JVM, but wanted a language that supported functional programming paradigms. Which do you prefer – Python or Clojure or something else? Comment below and let me know!

References

Visible links
1. https://docs.python.org/3/distutils/setupscript.html

HackerNewsBot debug: Calculated post rank: 110 - Loop: 154 - Rank min: 100 - Author rank: 35

On Learning Rust and Go: Migrating Away from Python

I first started using Python in 1993. It's been my main programming language since about 2000. I've written a ton of code in Python, both for work and in my free time. For several years now, I've been…
Article word count: 621

HN Discussion: https://news.ycombinator.com/item?id=19475218
Posted by edward (karma: 22125)
Post stats: Points: 119 - Comments: 90 - 2019-03-24T08:44:34Z

\#HackerNews #and #away #from #learning #migrating #python #rust

---

Article content:

I first started using Python in 1993. Itʼs been my main programming language since about 2000. Iʼve written a ton of code in Python, both for work and in my free time. For several years now, Iʼve been growing dissatisfied with it. Partly itʼs because Iʼd like more help from my programming tools, such as static type checking, better handling of abstractions and code modules, and in general aiding me in writing larger, more complex software. Partly itʼs because Iʼm writing more challenging software, and trying to get more out of the hardware I have available. Partly itʼs because Iʼm not getting the feeling that the Python community is going in a direction I want to follow. Instead I get the feeling that the Python community is happy to cut corners and compromise on things that Iʼm not willing to. Which is fine, if it makes their lives better, but leaves me wanting something else.

I wrote Obnam, my backup application, in Python, over a period of about fourteen years, until I retired it a year ago. During Obnamʼs life, Python 3 happened. Python 3 is actually a good thing, I think, but the transition was painful, and Obnam never made it. Obnam had other issues, which made it less fun to work on; Python 3 wasnʼt what killed it.

Obnam and other large programs Iʼve written in Python gave me the strong feeling that itʼs a nice language up to a certain size and complexity of program. Obnam is about 15000 lines of Python code. That turned out to be too much for me in Python: too often there were bugs that a static, strong type system would have caught. I could perhaps have been more diligent in how I used Python, and more careful in how I structured my code, but thatʼs my point: a language like Python requires so much self-discipline that at some point it gets too much.

So over the last few months Iʼve been learning Rust and Go, on and off, in short gaps of free time between other duties. Both have static type systems that can be argued to be strong. Both seem to have decent module systems. Both seem to support concurrency well. Either should be a good replacement for Python for non-small software I write. But I expect to be using Rust for any non-work programming and Go only when work needs me to.

Rust is developed by a community, and was started by Mozilla. Go development seems to be de facto controlled by Google, who originated the language. Iʼd rather bet my non-work future on a language that isnʼt controlled by a huge corporation, especially one of the main players in todayʼs surveillance economy. I write code in my free time because itʼs fun, and I release it as free software because thatʼs the ethical thing to do. I feel quite strongly that software freedom is one of the corner stones for the long-term happiness for humanity.

Anyway.

Ignoring ethical concerns, Rust seems like the better language of the two, so far. It has the better type system, the better compiler, the better tooling in general, and seems to me to have learnt better the lessons of programming languages and tools of the past third of a century. Rust exhibits better taste: things are designed the way they are for good reasons. Itʼs not always as convenient or familiar as Go, but it doesnʼt seem to make compromises for short-term convenience the way Go does.

Note that Iʼve not written any significant code in either language, so Iʼm just writing based on what Iʼve learnt by reading. My opinions may change in the future, as I get more into both languages.

HackerNewsBot debug: Calculated post rank: 109 - Loop: 78 - Rank min: 100 - Author rank: 74

The Boombox Incident: removing bald people from photos

Written by Matt Bilyeu on March 21, 2019 In Seinfeld episode #163, “The Slicer”, George has just landed a cushy job at Kruger Industrial Smoothing, when he sees himself in the background of a family…
Article word count: 850

HN Discussion: https://news.ycombinator.com/item?id=19462007
Posted by mbil (karma: 319)
Post stats: Points: 156 - Comments: 39 - 2019-03-22T12:47:20Z

\#HackerNews #bald #boombox #from #incident #people #photos #removing #the

---

Article content:



Written by Matt Bilyeu
on March 21, 2019

In Seinfeld episode [1]#163, “The Slicer”, George has just landed a cushy job at Kruger Industrial Smoothing, when he sees himself in the background of a family photo on his new boss’s desk.

[2]kruger-family-untouched

George is instantly reminded of the “boombox incident”, in which he had, years earlier, embarrassed himself in front of the Kruger family.

Later, upon hearing about the photo, Kramer suggests that George sneak the photo off to have himself airbrushed out of the picture, so that Kruger doesn’t remember the incident and fire George.

George enacts the plan and things are going fine until he receives the touched-up photo: the clerk has removed Kruger from the family photo instead of George.

[3]kruger_family_airbrushed

The clerk mistook Kruger in the photo for George, since in the picture George had hair but Kruger was bald. Removing the only bald person from the photo was a pretty reasonable thing for the photo store clerk to do. I figured this is something that photo editors have to do frequently, so I decided to automate it.

The process for removing bald people from photos is as follows:
\* detect faces in an image using off-the-shelf tools
\* for each face

\* roughly locate the forehead/hair region
\* get the dominant color of the face and of the top of the head
\* compare the two colors
\* consider a bald subject to be one where the two colors are very close, i.e. the top of the head is the same color as the face (skin tone)
\* attempt to inpaint the region containing the bald individual
Face detection

Face detection in OpenCV can be accomplished with a [4]cascade classifier. A pre-trained model from the [5]OpenCV data repository is made available. The underlying algorithm at play is the [6]Viola-Jones object detection framework, which uses a coarse-to-fine cascade of Haar-feature matching to identify human faces.

[7]haar_features

face_cascade = cv2.CascadeClassifier(ʼ./haarcascade_frontalface_default.xmlʼ) def detect_faces(image): h, w, _ = image.shape gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY) min_size = int(w*0.12) faces = face_cascade.detectMultiScale( gray, scaleFactor=1.1, minNeighbors=1, minSize=(min_size, min_size), flags = cv2.cv.CV_HAAR_SCALE_IMAGE ) return faces g_and_k_img = cv2.imread(ʼ./input_images/george-and-kruger.jpgʼ)
tmp = np.copy(g_and_k_img)
for (x, y, w, h) in detect_faces(g_and_k_img): cv2.rectangle(tmp, (x, y), (x+w, y+h), (0, 255, 0), 2)

[8]george_and_kruger_detection

[9]seinfeld_detection

Once the faces in a photo have been located, we need to examine just above each face region to determine whether the person is bald or has hair. This approach assumes, of course, that the faces are oriented vertically. A better method might be to locate the eyes and/or mouth using other cascade classifiers and then approximate the forehead position from there.

def forehead_region(img, face_loc): img_h, img_w = img.shape[:2] x, y, w, h = face_loc fore_w = int(w * 0.33) fore_h = int(h * 0.25) fore_x = min(x + ((w - fore_w) // 2), img_w) fore_y = max(y - fore_h, 0) return img[fore_y:fore_y + fore_h, fore_x:fore_x + fore_w, :]

[10]elaine_forehead_detail

Dominant color

The algorithm presented here compares the color in the forehead/top-of-head region to that of the face region. To find the dominant color, we can use [11]k-means clustering. This quantizes all pixel BGR values to be one of k colors, and we chose the most frequent. Simply averaging all colors in the image can also work well (and is faster) if the region is tightly bound.

def dominant_color(img): # via https://docs.opencv.org/3.0-beta/doc/py_tutorials/py_ml/py_kmeans/py_kmeans_opencv/py_kmeans_opencv.html
Z = img.reshape((-1,3)) Z = np.float32(Z) # define criteria, number of clusters(K) and apply kmeans()
criteria = (cv2.TERM_CRITERIA_EPS + cv2.TERM_CRITERIA_MAX_ITER, 200, 1.0) K = 3 _, labels, palette = cv2.kmeans( Z, K, criteria, 10, cv2.KMEANS_RANDOM_CENTERS ) # via https://stackoverflow.com/a/43111221
_, counts = np.unique(labels, return_counts=True) dominant = palette[np.argmax(counts)] return np.uint8(dominant)

[12]elaine_face_color [13]elaine_hair_color

Bald?

With the dominant color for each subject’s face and hair regions obtained, we can compare the values to determine baldness. [14]Comparing colors programatically is not as simple as taking the Euclidean distance of the BGR color vectors, because the resultant differences don’t correspond well to human color difference perception. A better distance metric is [15]ΔE* in the CIE Lab color space. I used the [16]colormath package to perform those distance calculations.

We compare the color difference to a threshold, and this heuristic is the baldness detection. It probably gives false positives for subjects with hair color close to their skin tone.

# via http://hanzratech.in/2015/01/16/color-difference-between-2-colors-using-python.html
from colormath.color_objects import sRGBColor, LabColor
from colormath.color_conversions import convert_color
from colormath.color_diff import delta_e_cie2000 def is_bald(img, face, thresh=40): face_img = face_region(img, face) forehead_img = forehead_region(img, face) face_color = dominant_color(face_img) forehead_color = dominant_color(forehead_img) face_rgb = sRGBColor(face_color[2], face_color[1], face_color[0]) fore_rgb = sRGBColor(forehead_color[2], forehead_color[1], forehead_color[0]) delta = delta_e_cie2000( convert_color(face_rgb, LabColor), convert_color(fore_rgb, LabColor) ) return delta < thresh

[17]low_delta [18]high_delta

[19]final_detection_0

[20]final_detection_1

[21]final_detection_2

Inpainting

Now that bald individuals have been identified, they can be removed from the image. OpenCV has an [22]inpainting function, which is really only meant for removing small strokes from an image. The results of applying it here are…not ideal.

def rm_bald(img): tmp = np.copy(img) img_h, img_w = img.shape[:2] for b in bald_locs(img): x, y, h, w = b img_h, img_w = tmp.shape[:2] mask = np.zeros(img[:,:,0].shape, np.uint8) mask[max(0, y-50):min(img_h, y+h), max(0, x-10):min(img_w, x+w+10)] = 1 mask[max(0, y+h):min(img_h, y+int(2.5*h)), max(0, x-10-w//2):min(img_w, x + w + int(w/2) + 10) ] = 1 tmp = cv2.inpaint(tmp, mask, 10, cv2.INPAINT_TELEA) return tmp

[23]inpainted_result

References:

References

Visible links
1.
4. https://docs.opencv.org/3.0-beta/doc/tutorials/objdetect/cascade_classifier/cascade_classifier.html
5. https://github.com/opencv/opencv/tree/master/data/haarcascades
6. file:///dev/frameworkhttps:/en.wikipedia.org/wiki/Viola%E2%80%93Jones_object_detection_framework#Cascade_architecture
11. https://docs.opencv.org/3.0-beta/doc/py_tutorials/py_ml/py_kmeans/py_kmeans_opencv/py_kmeans_opencv.html
14. http://hanzratech.in/2015/01/16/color-difference-between-2-colors-using-python.html
15. https://en.wikipedia.org/wiki/Color_difference#CIELAB_.CE.94E.2A
16. https://python-colormath.readthedocs.io/en/latest/
22. https://docs.opencv.org/3.3.1/df/d3d/tutorial_py_inpainting.html

HackerNewsBot debug: Calculated post rank: 117 - Loop: 217 - Rank min: 100 - Author rank: 58

Google hit with €1.5bn fine from EU over advertising

The search engine has been fined for blocking rival online search advertisers.
Article word count: 375

HN Discussion: https://news.ycombinator.com/item?id=19440926
Posted by okket (karma: 35450)
Post stats: Points: 169 - Comments: 140 - 2019-03-20T11:15:48Z

\#HackerNews #€15bn #advertising #fine #from #google #hit #over #with

---

Article content:




[1]Google logo Image copyright Getty Images Image caption Alphabetʼs Google remains a dominant force in online advertising

Google has been hit with a €1.49bn (£1.28bn) fine from the EU for blocking rival online search advertisers.

It is the third EU fine for the search and advertising giant in two years.

The case accuses Google of abusing its market dominance by restricting third-party rivals from displaying search ads between 2006 and 2016.

In response, Google changed its AdSense contracts with large third parties, giving them more leeway to display competing search ads.

Google owner Alphabet makes large amounts of money from advertising - [2]pre-tax profits reached $30.7bn (£23bn) in 2018, up from $12.66bn in 2017.

"Google has cemented its dominance in online search adverts and shielded itself from competitive pressure by imposing anti-competitive contractual restrictions on third-party websites.

"This is illegal under EU anti-trust rules," said EC commissioner Margrethe Vestager.

Last year, the EU competition authority [3]hit Google with a record €4.34bn fine for using its popular Android mobile operating system to block rivals.

This followed [4]a €2.42bn fine in 2017 for hindering rivals of shopping comparison websites.

AdSense fine

The European Commission said that websites often had an embedded search function.

When a consumer uses this, the website delivers both search results and search adverts, which appear alongside the search result.

Googleʼs "AdSense for search" product delivers those adverts for website publishers.

The Commission described Google as acting like "an intermediary, like an advertising broker".

In 2006, Google started to include "exclusivity clauses" in contracts which stopped publishers from placing ads from Google rivals such as Microsoft and Yahoo on search pages, the Commission said.

From 2009, Google started replacing the exclusivity clauses with "premium placement" clauses, which meant publishers had to keep the most profitable space on their search results pages for Googleʼs adverts and they had to request a minimum number of Google adverts.

Publishers also needed to get written permission from Google before making any changes to how rival ads were displayed, letting Google control "how attractive, and therefore clicked on, competing search adverts could be", the Commission said.

Between 2006 to 2016, Google had more than 70% of the search intermediation market in the EU. It generally had more than 90% of the search market and more than 75% of the online search advertising market, the Commission added.

References

Visible links
2. https://www.bbc.co.uk/news/business-47125043
3. https://www.bbc.co.uk/news/technology-44858238
4. https://www.bbc.co.uk/news/technology-40406542

HackerNewsBot debug: Calculated post rank: 159 - Loop: 64 - Rank min: 100 - Author rank: 69

KDE Connect removed from Google Play store for violating new policy on SMS

“KDE Connect has been removed from @GooglePlay for violating their new policy on apps that access SMS [1]. The policy has an explicit exception for companion apps (like KDE Connect), but it was…
Article word count: 590

HN Discussion: https://news.ycombinator.com/item?id=19429193
Posted by coolgoose (karma: 115)
Post stats: Points: 161 - Comments: 52 - 2019-03-19T09:09:29Z

\#HackerNews #connect #for #from #google #kde #new #play #policy #removed #sms #store #violating

---

Article content:



[1] [IMG][2]Albert Vaca Cintora‏ @albertvaka [3]3h3 hours ago
\* 
\* 
KDE Connect has been removed from [4]@GooglePlay for violating their new policy on apps that access SMS [1]. The policy has an explicit exception for companion apps (like KDE Connect), but it was removed anyway and thereʼs no way to talk to Google. 1/N [1][5]https://support.google.com/googleplay/android-developer/answer/9047303 …

[6] [IMG][7]Albert Vaca Cintora‏ @albertvaka [8]3h3 hours ago
\* 
\* 
Google only provides one-way forms to contact them. Iʼve filled the forms regarding this policy change (including one they sent to existing apps before the policy was effective) but never got an explanation to why KDE Connect doesnʼt qualify as a companion app. 2/N

Show this thread
[9] [IMG][10]Albert Vaca Cintora‏ @albertvaka [11]3h3 hours ago
\* 
\* 
There is simply no way to talk to a human being at [12]@Google. I bet this saves a lot in customer support. 3/N

Show this thread
[13] [IMG][14]Albert Vaca Cintora‏ @albertvaka [15]3h3 hours ago
\* 
\* 
Iʼm about to upload a version of KDE Connect to the Play Store with the SMS functionality removed. Those who want the full app, can still download it from F-Droid: [16]https://f-droid.org/packages/org.kde.kdeconnect_tp/ … Thanks [17]@fdroidorg! 4/N

Show this thread
[18] [IMG][19]Albert Vaca Cintora‏ @albertvaka [20]3h3 hours ago
\* 
\* 
To make things better, the [21]@GooglePlay console gives me an internal error, so I canʼt upload the version without SMS support. Meanwhile, new users canʼt download KDE Connect. [22]👏 Good job Google.[23]pic.twitter.com/Cy8D1CsOJH

Show this thread
[24] [IMG][25]Gürkan 🇪🇺�’‏ @gurkanfalan [26]2h2 hours ago
\* 
\* 
Replying to [27]@albertvaka [28]@GooglePlay

Thanks google for constantly reminding me that I should free myself from it.

[29] [IMG][30]Alfie Day‏ @Azelphur [31]2h2 hours ago
\* 
\* 
Replying to [32]@albertvaka [33]@GooglePlay

Good thing I was already using [34]@fdroidorg. [35]@GooglePlay needs sort this silliness out.

[36] [IMG][37]Andrei Costin Balaianu‏ @ABalaianu [38]3h3 hours ago
\* 
\* 
Replying to [39]@albertvaka [40]@GooglePlay

This is so Google (and Apple, and Facebook and...wait...I donʼt have enough characters available for this list)

[41] [IMG][42]Chris Hills‏ @chaz_6 [43]36m36 minutes ago
\* 
\* 
Replying to [44]@albertvaka [45]@GooglePlay

It would be useful if the *.apk files were available at [46]https://github.com/KDE/kdeconnect-android/releases/ … for direct installation in the meantime

[47] [IMG][48]mase‏ @mase_nocturnal [49]1h1 hour ago
\* 
\* 
Replying to [50]@albertvaka [51]@GooglePlay

Thankfully it is still on [52]@fdroidorg

[53] [IMG][54]Giuseppe Pignataro‏ @fastbyte01 [55]2h2 hours ago
\* 
\* 
Replying to [56]@albertvaka [57]@kubuntu [58]@GooglePlay

The Google support for developers publishing on [59]@GooglePlay is really worst. [60]@Android is the first mobile OS thanks for the support of all the developers that create apps for it and they arenʼt able to give us a decent support.

[61] [IMG][62]Carlos Soriano‏ @csoriano1618 [63]53m53 minutes ago
\* 
\* 
Replying to [64]@albertvaka [65]@GooglePlay

Recently we had some issues too with Google API policies at GNOME... itʼs not being easy to work effectively with them [66]😔

[67] [IMG][68]Carlos Soriano‏ @csoriano1618 [69]51m51 minutes ago
\* 
\* 
Replying to [70]@csoriano1618 [71]@albertvaka [72]@GooglePlay

This is like all big companies though, we had a good experience with other departments like GSoc or so.

[73] [IMG][74]RAMY‏ @RamySami [75]2m2 minutes ago
\* 
\* 
Replying to [76]@albertvaka [77]@GooglePlay

Epic was right not to put Fortnite on Google play

[78] [IMG][79]RTheren‏ @r_theren [80]4m4 minutes ago
\* 
\* 
Replying to [81]@albertvaka [82]@GooglePlay

I use the version fro F-Droid anyway. This really sucks, like a lot.

[83] [IMG][84]Chris Hills‏ @chaz_6 [85]38m38 minutes ago
\* 
\* 
Replying to [86]@albertvaka [87]@GooglePlay

Perhaps this could be pursued under antitrust laws? They must be able to be held to account.

[88] [IMG][89]Anthony Maurin‏ @spartanz51 [90]54m54 minutes ago
\* 
\* 
Replying to [91]@albertvaka [92]@Alex193a [93]@GooglePlay

@GooglePlay please fix that

[94] [IMG][95]Emrah Urhan‏ @raxetul [96]1h1 hour ago
\* 
\* 
Replying to [97]@albertvaka [98]@sinanonur [99]@GooglePlay

[100]😡

[101] [IMG][102]Vladymyr L‏ @Vladymyr_L [103]2h2 hours ago
\* 
\* 
Replying to [104]@albertvaka [105]@GooglePlay

I think the worst thing that could do Google for his customer is to leave them without the best program without any real reason just for some dumb policy.

References

Visible links
1. https://twitter.com/albertvaka
2. https://twitter.com/albertvaka
3. https://twitter.com/albertvaka/status/1107924633750253568
4. https://twitter.com/GooglePlay
5. https://t.co/WDXEqTUhHl
6. https://twitter.com/albertvaka
7. https://twitter.com/albertvaka
8. https://twitter.com/albertvaka/status/1107924634823987200
9. https://twitter.com/albertvaka
10. https://twitter.com/albertvaka
11. https://twitter.com/albertvaka/status/1107924635818098689
12. https://twitter.com/Google
13. https://twitter.com/albertvaka
14. https://twitter.com/albertvaka
15. https://twitter.com/albertvaka/status/1107924636799504384
16. https://t.co/vsudDrzzgS
17. https://twitter.com/fdroidorg
18. https://twitter.com/albertvaka
19. https://twitter.com/albertvaka
20. https://twitter.com/albertvaka/status/1107924637927772160
21. https://twitter.com/GooglePlay
23. https://t.co/Cy8D1CsOJH
24. https://twitter.com/gurkanfalan
25. https://twitter.com/gurkanfalan
26. https://twitter.com/gurkanfalan/status/1107944301487706112
27. https://twitter.com/albertvaka
28. https://twitter.com/GooglePlay
29. https://twitter.com/Azelphur
30. https://twitter.com/Azelphur
31. https://twitter.com/Azelphur/status/1107942514982555648
32. https://twitter.com/albertvaka
33. https://twitter.com/GooglePlay
34. https://twitter.com/fdroidorg
35. https://twitter.com/GooglePlay
36. https://twitter.com/ABalaianu
37. https://twitter.com/ABalaianu
38. https://twitter.com/ABalaianu/status/1107933048375595010
39. https://twitter.com/albertvaka
40. https://twitter.com/GooglePlay
41. https://twitter.com/chaz_6
42. https://twitter.com/chaz_6
43. https://twitter.com/chaz_6/status/1107965389470879745
44. https://twitter.com/albertvaka
45. https://twitter.com/GooglePlay
46. https://t.co/xDeIT4JimH
47. https://twitter.com/mase_nocturnal
48. https://twitter.com/mase_nocturnal
49. https://twitter.com/mase_nocturnal/status/1107955556042473474
50. https://twitter.com/albertvaka
51. https://twitter.com/GooglePlay
52. https://twitter.com/fdroidorg
53. https://twitter.com/fastbyte01
54. https://twitter.com/fastbyte01
55. https://twitter.com/fastbyte01/status/1107951448304898048
56. https://twitter.com/albertvaka
57. https://twitter.com/kubuntu
58. https://twitter.com/GooglePlay
59. https://twitter.com/GooglePlay
60. https://twitter.com/Android
61. https://twitter.com/csoriano1618
62. https://twitter.com/csoriano1618
63. https://twitter.com/csoriano1618/status/1107961030204899328
64. https://twitter.com/albertvaka
65. https://twitter.com/GooglePlay
67. https://twitter.com/csoriano1618
68. https://twitter.com/csoriano1618
69. https://twitter.com/csoriano1618/status/1107961542547517441
70. https://twitter.com/csoriano1618
71. https://twitter.com/albertvaka
72. https://twitter.com/GooglePlay
73. https://twitter.com/RamySami
74. https://twitter.com/RamySami
75. https://twitter.com/RamySami/status/1107973938615906307
76. https://twitter.com/albertvaka
77. https://twitter.com/GooglePlay
78. https://twitter.com/r_theren
79. https://twitter.com/r_theren
80. https://twitter.com/r_theren/status/1107973513917411328
81. https://twitter.com/albertvaka
82. https://twitter.com/GooglePlay
83. https://twitter.com/chaz_6
84. https://twitter.com/chaz_6
85. https://twitter.com/chaz_6/status/1107964950243393541
86. https://twitter.com/albertvaka
87. https://twitter.com/GooglePlay
88. https://twitter.com/spartanz51
89. https://twitter.com/spartanz51
90. https://twitter.com/spartanz51/status/1107960807386685445
91. https://twitter.com/albertvaka
92. https://twitter.com/Alex193a
93. https://twitter.com/GooglePlay
94. https://twitter.com/raxetul
95. https://twitter.com/raxetul
96. https://twitter.com/raxetul/status/1107958422920720385
97. https://twitter.com/albertvaka
98. https://twitter.com/sinanonur
99. https://twitter.com/GooglePlay
101. https://twitter.com/Vladymyr_L
102. https://twitter.com/Vladymyr_L
103. https://twitter.com/Vladymyr_L/status/1107948467480154112
104. https://twitter.com/albertvaka
105. https://twitter.com/GooglePlay

HackerNewsBot debug: Calculated post rank: 124 - Loop: 108 - Rank min: 100 - Author rank: 82

EU government websites have undisclosed adtech trackers from Google and others

Plus: UK health service sites contain commercial trackers
Article word count: 857

HN Discussion: https://news.ycombinator.com/item?id=19424041
Posted by snaky (karma: 3287)
Post stats: Points: 162 - Comments: 44 - 2019-03-18T19:03:20Z

\#HackerNews #adtech #and #from #google #government #have #others #trackers #undisclosed #websites

---

Article content:




All but three of the European Union member statesʼ government websites are littered with undisclosed adtech trackers from Google and other firms, with many piggy-backing on third-party scripts, according to an analysis of almost 200,000 webpages.

The [1]report (PDF), published today by Cookiebot in collaboration with civil rights association European Digital Rights (EDRi), scanned 184,683 EU government webpages on 11 and 12 March to assess the cookies on each.

It found that there were 112 companies slurping up information on EU citizensʼ browsing habits on the webpages of the governments supposedly fighting the good fight against excess stalking of netizens.

Adtech trackers were found on 25 of the 28 member statesʼ sites, with only Spain, Germany and the Netherlands clean of commercial cookies. There were 52 companies identified on Franceʼs government sites, 27 on Latviaʼs and 19 on Belgiumʼs. Twenty cookies were [2]identified on GOV.UK, of which 12 were marketing, and all belonged to one company – Google.

Indeed, the search giant is described as the "kingpin of tracking" within the report, present on 82 per cent of all the sites and accounting for three of the top five trackers: YouTube, DoubleClick and Google.

The report authors said this was of "special concern" because Google can cross-reference trackers with its first-party account details via its widely used consumer services such as Mail, Search and Android apps.

Separately, the work assessed public health service sites, again finding that cookies were widespread, with 52 per cent of those tested having commercial trackers.

And again, Google was right up there, making up two of the top five, with the others being Adobeʼs eversttech.net, AppNexusʼ adnxs.com and Mediamathʼs Mathtag.com.

For this assessment, the researchers chose six EU countries and carried out 15 health-related search queries – such as "How do I know if I have HIV?", "Signs of being an alcoholic" and "I want to terminate my pregnancy" – from IP addresses in each country to identify the relevant landing pages on each nationʼs health service.

In the UK, some 60 per cent of these landing pages had such ad trackers, less only than Irish sites, where trackers appeared on 73 per cent of landing pages. A single German website about maternity leave was monitored by 63 companies, while a French page about abortion was tracked by 21 firms.

The group said this could be used to "infer sensitive facts about [usersʼ] health condition and life situation" and be resold to target ads. "These citizens have no clear way to prevent this leakage, understand where their data is sent, or to correct or delete the data," it said.

The extent of tracking on these sites is even more alarming, the report argued, because they donʼt rely on ad revenue. In some cases, governments will want to use companiesʼ services, but in others the firms gained access to these non-commercial sites through "free" third-party JavaScript tech services, like share buttons or plugins.

"These scripts can act as Trojan horses, opening backdoors to the website code through which ad tech companies can silently insert their trackers," the report said.

It urged website owners to be more careful when including third-party components on their sites; to make sure they had a detailed overview of the current trackers; and to remove any unwanted ones from the source code.

Visitors should also be offered full transparency and control over trackers on the site – but it shouldnʼt just be up to users to lock down their browsing habits. Stronger regulations need to be in force, and adhered to.

"How can any organisation live up to its [European General Data Protection Regulation] GDPR and ePrivacy obligations if it does not control unauthorised tracking actors accessing their website?" asked Cookiebot founder Daniel Johannsen.

"Public sector bodies now have the opportunity to lead by example – at a minimum by shutting down any digital rights infringements that they are facilitating on their own websites."

Diego Naranjo at EDRi used the opportunity to lament the delay to the long-awaited [3]ePrivacy Regulation, which was initially meant to be enforced as the yin to the GDPRʼs yang, covering communications data rather than personal data.

However, it has been [4]stuck in discussions between member states for more than a year, and privacy activists fear it is being watered down as a result of lobbying from adtech industry and concerns among member states.

If it does lose ground, Naranjo warned, it will "open a Pandoraʼs box of more and more sharing, merging and reselling of personal data in huge online commercial surveillance networks, in which citizens are being unwittingly tracked and micro-targeted with commercial and political manipulation."

Their calls for progress echo those made by the European Data Protection Board last week. The group – made up of the blocʼs data protection watchdogs and EU data protection supervisor – issued a statement urging legislators to "intensify efforts" to adopt it.

"The future ePrivacy Regulation should under no circumstance lower the level of protection offered by the current ePrivacy Directive and should complement the GDPR by providing additional strong guarantees for all types of electronic communications," it [5]said. ®

Sponsored: [6]Becoming a Pragmatic Security Leader

References

Visible links
1. https://www.cookiebot.com/media/1121/cookiebot-report-2019-medium-size.pdf
2. https://www.dropbox.com/sh/uz5lrz5bqglracj/AAD9s8fzdpZem1KE-1KvNKqFa/UK?dl=0&subfolder_nav_tracking=1
3. https://www.theregister.co.uk/2017/12/11/european_parliament_eprivacy_rapporteur_birgit_sippel_interview/
4. https://www.theregister.co.uk/2019/01/02/2019_eprivacy_brexit/
5. https://edpb.europa.eu/news/news/2019/european-data-protection-board-eighth-plenary-session-interplay-eprivacy-directive_en
6. https://go.theregister.co.uk/tl/1818/-7142/becoming-a-pragmatic-security-leader?td=wptl1818

HackerNewsBot debug: Calculated post rank: 122 - Loop: 125 - Rank min: 100 - Author rank: 24

Italy bans unvaccinated children from school

Italy now requires children to prove they have been vaccinated before attending school.
Article word count: 711

HN Discussion: https://news.ycombinator.com/item?id=19423147
Posted by hackr_nj (karma: 75)
Post stats: Points: 124 - Comments: 87 - 2019-03-18T17:35:23Z

\#HackerNews #bans #children #from #italy #school #unvaccinated

---

Article content:




[1]A generic photograph of a vaccine being drawn from a vial into a needle Image copyright Getty Images Image caption The new law demands 10 compulsory vaccinations - and has proved controversial

Italian children have been told not to turn up to school unless they can prove they have been properly vaccinated.

The deadline follows months of national debate over compulsory vaccination.

Parents risk being fined up to €500 (£425; $560) if they send their unvaccinated children to school. Children under six can be turned away.

The new law came amid a surge in measles cases - but Italian officials say vaccination rates have improved since it was introduced.

Under Italyʼs so-called Lorenzin law - named after the former health minister who introduced it - children must receive a range of mandatory immunisations before attending school. They include vaccinations for chickenpox, polio, measles, mumps, and rubella.

Children up to the age of six years will be excluded from nursery and kindergarten without proof of vaccination under the new rules.

Those aged between six and 16 cannot be banned from attending school, but their parents face fines if they do not complete the mandatory course of immunisations.

The deadline for certification was due to be 10 March after a previous delay - but as it fell on a weekend, it was extended to Monday.

"Now everyone has had time to catch up," Health Minister Giulia Grillo told La Repubblica newspaper.

She had reportedly resisted political pressure from deputy prime minister Matteo Salvini to extend the deadline even further.

Ms Grillo said the rules were now simple: "No vaccine, no school".

Italian media report that regional authorities are handling the situation in a number of different ways.

In Bologna, the local authority has sent letters of suspension to the parents of some 300 children, and a total of 5,000 children do not have their vaccine documentation up to date.

In other areas there have been no reported cases, while still others have been given a grace period of a few days beyond the deadline.

Is the law having an effect?

The new law was passed to raise Italyʼs plummeting vaccination rates from below 80% to the World Health Organisationʼs 95% target.

On Monday - the last day for parents to provide documentation proving their children had been properly vaccinated - the Italian health authority released figures claiming a national immunisation rate at or very close to 95% for children born in 2015, depending on which vaccine was being discussed.

The 95% threshold is the point at which "herd immunity" kicks in - when enough of the population is vaccinated for the spread of the disease to become unlikely, thereby protecting those who cannot be vaccinated.

That includes babies too young to be vaccinated themselves, or those with medical conditions such as a compromised immune system.

Last month, an eight-year-old recovering from cancer was unable to attend school in Rome due to his weak immune system.

The child had spent months receiving treatment for leukaemia, but was at risk of infection because a proportion of pupils in the school had not been vaccinated - including several in the same class.

Image copyright Getty Images
Image caption Demonstrations against compulsory vaccination were held in Rome, 2017

The Lorenzin law, drafted by the previous government, had a tumultuous birth. When the current coalition came to power, it said it would drop mandatory immunisations although it later reversed its position.

The two populist parties in power had faced accusations that they were pursuing anti-vaccination policies.

Writing in a Facebook post on Monday, Ms Grillo admitted it "is a law that, at the time of approval, we criticised for several reasons" - and said that the law would be changed to include only those vaccinations that were necessary based on scientific data.

Why do parents not immunise their children?

The anti-vaccination movement has been growing globally in recent years, sparking alarm from the World Health Organization.

A long-discredited paper by Andrew Wakefield was behind much of the scare, but rumours around immunisation have continued to spread, leading to public health risks as not enough people are immune to such diseases.

Mr Wakefield was struck off the UK medical register after fraudulently claiming there was a link between the measles, mumps and rubella vaccine (MMR) and autism and bowel disease in children.

He made the claim based on the experiences of just 12 children, and no other study since has been able to replicate his results.

References

Visible links

HackerNewsBot debug: Calculated post rank: 111 - Loop: 113 - Rank min: 100 - Author rank: 93

Americans Are Going Bankrupt from Getting Sick

Doctors’ bills play a role in 60 percent of personal-bankruptcy filings.

HN Discussion: https://news.ycombinator.com/item?id=19409244
Posted by clumsysmurf (karma: 4826)
Post stats: Points: 87 - Comments: 118 - 2019-03-16T18:11:46Z

\#HackerNews #americans #are #bankrupt #from #getting #going #sick

---

Article content:




Since Chamberlin Edmonds was offering only to help her find government insurance, for which Lockett did not qualify, the company wasn’t able to reduce her hospital bill or work out a payment plan. Lockett assumed this meant she was at a dead end with her nearly $30,000 in hospital bills. “They were saying that there was nothing that they could do,” she told me. She decided to put the bills aside and try to get to them after she found a job.

A year later, Lockett attended a meeting with representatives of Georgia Watch, a nonprofit group, as part of her volunteer work. When one of the Georgia Watch representatives mentioned that the organization has a [1]guide for people who wish to negotiate down their hospital bills, Lockett grabbed a copy. She called the hospital back. This time, she says, they told her her entire bill had been wiped away.

Emory Healthcare told me it could not comment on individual patients, but added, “Patients will sometimes receive two bills for the same date of service: one bill for services rendered by the physician; the other for a hospital stay, supplies, services, and equipment provided. Emory Healthcare’s customer-service department works with patients to establish a mutually acceptable agreement for paying inpatient or outpatient bills.”

[2]Read: Even the insured often can’t afford their medical bills

“The reality is that medical costs are not objective, real costs,” says Berneta L. Haynes, the director of equity and access at Georgia Watch. One day, an MRI can cost $19,000. The next, it can cost nothing.

Though she was still responsible for her reduced ambulance bill, Lockett was lucky. Others aren’t. Dana Peterman, a physical therapist in Forsyth, Georgia, owed more than $4,000, after insurance, when her son was rushed to the hospital with an anaphylactic peanut-allergy reaction in 2017. She tried to negotiate down her bill, but she says the hospital, the ambulance company, and the ER doctors did not give her a discount. She paid in full, not wanting the bills to affect her credit.

To negotiate with a hospital, consumer advocates I spoke with recommended asking about financial assistance, including charity care for the uninsured. If that fails, patients can ask whether they can pay whatever the hospital would have charged someone who was on Medicare—typically a lower rate. Hospitals and even collections agencies will often agree to payment plans, or a discount in exchange for a lump-sum payment.

Still, the current system requires people to independently negotiate on their own behalf with giant corporations over tens of thousands of dollars, often while recovering from a major illness. For those who haven’t done it before, the process can be confounding. “Maybe I didn’t say the right thing before,” Lockett told me.

If the patient fails to pay, a medical debt might be sent to a debt collector. Some patient advocates say small medical debts are now getting sold to debt buyers, companies that try to collect as much as they can on long-past-due debts. “Now we are seeing small-time medical practices get involved in selling their bad debts to debt buyers for pennies on the dollar,” says Sandoval-Moshenberg, of the Legal Aid Justice Center. Rather than medical records or a patient history, these buyers rely on little more than a list of debts in a spreadsheet, making it harder, Sandoval-Moshenberg and others argue, for patients to negotiate a deal or expunge an error. But this practice makes financial sense for doctors, given how many people are unable to pay their bills.

When everything fails, and the person is at imminent risk of having wages garnished because they’ve been sued for medical debt, it might be time to file for bankruptcy, says Sandoval-Moshenberg. The people who do become the tip of a very big debt iceberg.

We want to hear what you think about this article. [3]Submit a letter to the editor or write to letters@theatlantic.com.
[4]
[IMG][5]Olga Khazan is a staff writer at The Atlantic.
References

Visible links
1. https://www.georgiawatch.org/wp-content/uploads/2018/11/Georgia-Consumer-Guide-for-Medical-Bills-and-Debt_web_final_updated.pdf
2. https://www.theatlantic.com/business/archive/2017/06/medical-bills/530679/
3. https://www.theatlantic.com/contact/letters/
4. https://www.theatlantic.com/author/olga-khazan/
5. https://www.theatlantic.com/author/olga-khazan/

HackerNewsBot debug: Calculated post rank: 97 - Loop: 314 - Rank min: 80 - Author rank: 68

Proton Technologies awarded €2M from the EU

We are happy to announce that the European Commission has awarded €2 million to Proton Technologies AG, the parent company of ProtonMail and ProtonVPN, to develop a suite of encrypted services. Proton…

HN Discussion: https://news.ycombinator.com/item?id=19344937
Posted by dotcoma (karma: 1935)
Post stats: Points: 102 - Comments: 49 - 2019-03-09T07:15:09Z

\#HackerNews #€2m #awarded #from #proton #technologies #the

---

Article content:




[1]protonmail-blog-eu-funding

Proton Technologies has always been engaged in research as a legacy of [2]our scientific past. Whether it’s designing [3]open source cryptographic libraries, modernizing existing standards, or developing new standards, research and development has always comprised a large fraction of our expenditures. We are glad to announce that the European Commission’s [4]Horizon 2020 program, which has distributed nearly €80 billion across Europe to encourage scientific research and technological innovation, has recognized our contributions to the European economy and will be granting us €2 million to further our mission.

Switzerland, despite being outside of the EU, qualifies for Horizon 2020 funding because of bilateral agreements between Switzerland and the EU. The EU has always represented an integral part of our community, accounting for approximately 40% of ProtonMail users. It is also a core part of who we are: outside of our headquarters in Switzerland, 20% of our workforce is based in our offices in Czech Republic and Lithuania, and EU citizens collectively represent approximately half of our workforce.

Horizon 2020 in many ways aligns with our vision. First, Horizon 2020 funding comes in the form of grants, and because of that, they do not increase our debt, nor does it give the EU any shares or voting control in Proton Technologies. Control of the company therefore remains with our team, allowing us to continue putting the priorities of the community first. This grant also does not create any commitments on our part, other than using the funding for the purposes that we have outlined in our proposal to the European Commission. It does not alter our Swiss jurisdiction, nor can it compel us to do anything that would compromise the security of our product and privacy of our users in any way.

Second, this funding aligns the EU with our long-term success. The EU has now clearly signaled its support for privacy rights and is betting on Proton Technologies to deliver a European answer to the data mining monopolies of US tech giants like Google. In summary, we have not just gained a meaningful amount of funding, but also a powerful ally in the tough battles to come.

The Horizon 2020 selection process was rigorous. For over a year, the European Commission thoroughly scrutinized our team, our technology, our financials, and our main stakeholders during multiple rounds of evaluation, including a visit to Brussels to present Proton Technologies to European Commission officials. In this process, the Commission also sought input from the people who use our software, including some of the millions of individual EU citizens who use ProtonMail, and also customers of our [5]business secure email solutions.

We are glad that the European Commission has recognized our technical excellence, transparent and ethical business model, and the importance of our mission. At a time when many governments are looking to roll back privacy protections, sometimes with the collusion of massive tech giants, we are heartened that the EU is following through on the commitment to privacy set out by the [6]GDPR, and is now investing real money into privacy tech for the first time.

The European Commission has specific requirements for the use of funds, and this funding is mostly earmarked for building technologies for ProtonDrive, as the EU would like to see an expansion of the Proton product offerings to enhance our global competitiveness. While €2 million may not look like much in the big scheme of what we are trying to accomplish, every bit helps. This funding will without a doubt accelerate our ProtonDrive efforts and help us bring more value to the existing Proton community.

We also believe this is the right choice for Europe. Today, the tech industry is dominated by a small number of US tech giants, who exercise near monopoly power over the data of European citizens, controlling what we see online and how we get online, with a complete disregard for privacy. A European alternative that puts privacy first is not only a win for Europe, but of great benefit to the world.

Disrupting any industry and taking on gigantic competitors is always a challenge. That’s why having the support of the European Commission in this endeavor is a big win for the Proton community. We now have the chance to build a different and better Internet, one that puts user rights first, and one that seeks to improve the lives of all, and not just the few.

Best Regards,

The ProtonMail Team

You can get a [7]free secure email account from ProtonMail here.

We also provide a [8]free VPN service to protect your privacy.

ProtonMail and ProtonVPN are funded by community contributions. If you would like to support our development efforts, you can upgrade to a [9]paid plan or [10]donate. Thank you for your support.

Share This![11]Share on Google+[12]Share on Reddit[13]Share on Facebook[14]Email this to someone[15]Tweet about this on Twitter[16]Get a Free Encrypted Email Account

References

Visible links
2. http://protonmail.com/about
3. https://protonmail.com/blog/openpgpjs-4-streaming-encryption/
4. https://ec.europa.eu/programmes/horizon2020/en
5. http://protonmail.com/business
6. http://gdpr.eu/
7. https://protonmail.com/signup
8. https://protonvpn.com/
9. https://protonmail.com/support/knowledge-base/paid-plans/
10. https://protonmail.com/donate
11. https://plus.google.com/share?url=https://protonmail.com/blog/eu-funding/
12. http://reddit.com/submit?url=https://protonmail.com/blog/eu-funding/&title=We%20have%20been%20awarded%20€2%20million%20from%20the%20EU%20to%20further%20develop%20the%20Proton%20ecosystem
13. http://www.facebook.com/sharer.php?u=https://protonmail.com/blog/eu-funding/
14. mailto:?Subject=We%20have%20been%20awarded%20€2%20million%20from%20the%20EU%20to%20further%20develop%20the%20Proton%20ecosystem&Body=%20https://protonmail.com/blog/eu-funding/
15. http://twitter.com/share?url=https://protonmail.com/blog/eu-funding/&text=We+have+been+awarded+%E2%82%AC2+million+from+the+EU+to+further+develop+the+Proton+ecosystem+
16. https://protonmail.com/signup

HackerNewsBot debug: Calculated post rank: 84 - Loop: 39 - Rank min: 80 - Author rank: 27

Updates from YC

We have a few changes about YC to share.
Article word count: 561

HN Discussion: https://news.ycombinator.com/item?id=19342184
Posted by peteratt (karma: 241)
Post stats: Points: 148 - Comments: 60 - 2019-03-08T21:06:59Z

\#HackerNews #from #updates

---

Article content:




We have a few updates about YC to share.

Sam is transitioning to Chairman of YC and has shifted his operational responsibilities at YC to other partners. This change will allow Sam to spend more time focusing on [1]OpenAI while still being responsible, along with the rest of the partnership, for the long-term social and economic success of YC. Because YC is run as a partnership, there will be no significant operational change.

Things are going well — in the past two years, we launched [2]Startup School, [3]the Series A program, [4]the YC Growth program, [5]Work at a Startup, and [6]YC China. The current total market capitalization of YC companies is around $150 billion. The YC network now has over 4,000 alumni and 1,900 companies, and it has become a reliable source of advice, customers, friendship, and support for YC founders.

We had over 12,000 applicants for the Winter 2019 batch. This was a 30% increase compared to the Summer 2018 batch. We had two choices—either we reject a lot of great founders, or we figure out how to grow the batch. We decided to grow. We funded over 200 companies this batch, an increase of 25% compared to S18.

That necessitated some changes to [7]Demo Day. We will now have two stages running in parallel over two days, allowing more time for investors to interact with founders off-stage. With a larger space, we’ve been able to invite many more investors so that investment firms can send more people. (For those investors who want to watch every pitch, we record the presentations and make them available to watch on our Demo Day website.) Because of these changes, we had to move to a new venue. We had been talking for some time about moving to San Francisco, and this seemed like the natural time to do it.

Speaking of San Francisco, we are considering moving YC to the city and are currently looking for space. The center of gravity for new startups has clearly shifted over the past five years, and although we love our space in Mountain View, we are rethinking whether the logistical tradeoff is worth it, especially given how difficult the commute has become. We also want to be closer to our Bay Area alumni, who disproportionately live and work in San Francisco.

At the same time, we’ve gotten better at serving and being more accessible to founders who live around the world. Startup School is our free online program to teach people how to start a startup and to provide advice and community. Last year we had 10,500 companies participate. We’re particularly proud that 30% of the companies in this current batch came from Startup School! Our mission with this program is to bring the cost of knowledge, community, and tools to start a startup way down. Going forward, everyone who wants to do Startup School will be accepted.

Finally, we continue to scale up our Series A program, which helps our companies raise an A round when they’re ready. In the last twelve months, there were 96 Series As across the YC portfolio, and the Series A program helped 67 of those companies raise their rounds. We also started running Series A batches, modeled after the classic YC batches, and have created an investor portal to help more investors to get access to YC Series As.

References

Visible links
1. https://openai.com/
2. https://www.startupschool.org/
3. https://blog.ycombinator.com/introducing-yc-series-a-batches/
4. https://blog.ycombinator.com/growth-stage-program/
5. https://www.workatastartup.com/
6. https://blog.ycombinator.com/yc-china-qi-lu/
7. https://www.ycombinator.com/demoday/faq/

HackerNewsBot debug: Calculated post rank: 118 - Loop: 82 - Rank min: 100 - Author rank: 41

China bans 23m from buying travel tickets as part of 'social credit' system

People accused of social offences blocked from booking flights and train journeys
Article word count: 405

HN Discussion: https://news.ycombinator.com/item?id=19281192
Posted by EastToWest (karma: 70)
Post stats: Points: 128 - Comments: 144 - 2019-03-01T15:00:44Z

\#HackerNews #23m #bans #buying #china #credit #from #part #social #system #tickets #travel

---

Article content:




[1]China blocked 23 million “discredited” travellers from buying plane or train tickets last year as part of the country’s controversial “social credit” system aimed at improving the behaviour of citizens.

According to the National Public Credit Information Centre’s 2018 report, 17.5 million people were banned from buying flights and 5.5 million barred from purchasing high-speed train tickets because of social credit offences. The report released last week said: “Once discredited, limited everywhere”.

The social credit system aims to incentivise “trustworthy” behaviour through penalties as well as rewards. According to a government document about the system dating from 2014, the aim is to “allow the trustworthy to roam everywhere under heaven while making it hard for the discredited to take a single step.”

Social credit offences range from not paying individual taxes or fines to spreading false information and taking drugs. More minor violations include using expired tickets, smoking on a train or not walking a dog on a leash.

Local governments and agencies have been piloting aspects of the system, which will eventually give every Chinese citizen a personalised score. Critics saidauthorities in China were using technology and big data to create an Orwellian state of mass surveillance and control.

Authorities have previously used blacklists to limit the travel of some citizens, but the social credit system appears to have expanded the practice. China’s supreme court said in 2017 that 6.15 million citizens had been barred from taking flights because of social credit offences.

Last year, China’s National Development and Reform Commission said it would begin banning people on public transport for up to a year. The recent report also said 128 people were prevented from leaving China because of unpaid taxes.

According to the report, other penalties for individuals include being barred from buying insurance, real estate or investment products. Companies on the blacklist are banned from bidding on projects or issuing corporate bonds.

The report said authorities collected more than 14m data points of “untrustworthy conduct” last year, including scams, unpaid loans, false advertising and occupying reserved seats on a train.

A video of a train passenger who refused to give up a seat another passenger had reserved went viral last year, sparking a debate over whether the current system of banning such travellers was harsh enough.

Officials believe the system is working. The report said 3.5 million people or companies paid taxes or debts they owed because of the social credit system.

References

Visible links
1. https://www.theguardian.com/world/china

HackerNewsBot debug: Calculated post rank: 133 - Loop: 124 - Rank min: 100 - Author rank: 87

America’s Cities Are Running on Software from the ’80s

America’s Cities Are Running on Software From the ’80s
Article word count: 1099

HN Discussion: https://news.ycombinator.com/item?id=19271569
Posted by eplanit (karma: 10893)
Post stats: Points: 82 - Comments: 103 - 2019-02-28T14:42:15Z

\#HackerNews #80s #americas #are #cities #from #running #software #the

---

Article content:




(Bloomberg Businessweek) -- The only place in San Francisco still pricing real estate like it’s the 1980s is the city assessor’s office. Its property tax system dates back to the dawn of the floppy disk. City employees appraising the market work with software that runs on a dead programming language and can’t be used with a mouse. Assessors are prone to make mistakes when using the vintage software because it can’t display all the basic information for a given property on one screen. The staffers have to open and exit several menus to input stuff as simple as addresses. To put it mildly, the setup “doesn’t reflect business needs now,” says the city’s assessor, Carmen Chu.

San Francisco rarely conjures images of creaky, decades-old technology, but that’s what’s running a key swath of its government, as well as those of cities across the U.S. Politicians can often score relatively easy wins with constituents by borrowing money to pay for new roads and bridges, but the digital equivalents of such infrastructure projects generally don’t draw the same enthusiasm. “Modernizing technology is not a top issue that typically comes to mind when you talk to taxpayers and constituents on the street,” Chu says. It took her office almost four years to secure [1]$36 million for updated assessors’ hardware and software that can, among other things, give priority to cases in which delays may prove costly. The design requirements are due to be finalized this summer.

For local officials throughout the country, the shift from old-school servers to rented cloud storage has made it tougher than ever to fund upgrades. They can budget physical equipment as capital expenses, meaning they could issue bonds to pay for them. But cloud computing is a service, as the people selling it love to say, which means officials have to pay for it with operating funds—the same pool of money that goes toward addressing more tangible demands, such as parks and cops. The deliberate pace of government compounds the problem of strained resources, says [2]Marc Pfeiffer, a former New Jersey official who now advises municipalities on managing technology as part of Rutgers University’s Bloustein Local Government Research Center.

Businesses have started to see municipal woes as opportunities. Salesforce.com Inc., a maker of cloud-based applications that are being put to work on the assessor’s office project in the company’s hometown of San Francisco, has focused more on government solutions in the past five years. During an [3]earnings call in August, co-Chief Executive Officer Keith Block called the public sector “one of our strongest verticals.” But cities and states are continuing to plow money into maintaining legacy systems, because they still work. Upgrades are difficult to start—and to finish—and attracting new skilled workers is expensive.

The impetus for change is often public outcry over a crisis, such as the [4]chaotic 2009 crash of a disco-era computer system regulating traffic signals in Montgomery County, Md., or the cyberattacks that brought Atlanta’s government to a standstill last March. And promises to improve are no guarantee of success: Minnesota spent about a decade and $100 million to replace its ancient vehicle-licensing and registration software, but the new version arrived with so many glitches in 2017 that Governor Tim Walz has asked for an [5]additional $16 million to fix it.

Of course, improvements cost money that constituents don’t always want to pay. “We’re dealing with an irrational public who wants greater and greater service delivery at the same time they want their taxes to be lower,” says [6]Alan Shark, executive director of the Public Technology Institute, an association for municipal tech officials.

In San Francisco the assessor uses a Cobol-based system called AS-400, whose welcome screen reads, “COPYRIGHT IBM CORP., 1980, 2009.” As the [7]city tax rolls jumped 22 percent over two years, workers were struggling to keep track of the changes on their ancient systems. At one point they fell three years behind. It’s a “lot of manual work” just to perform basic functions, Chu says.

Searches that should seem simple take much longer because of the system’s quirks. If a resident contacts the agency saying her house should have a different assessed value, a worker has to look up the block and identification number that’s technically taxed; there’s no way to filter by address. Also, all street numbers need to have four digits, so 301 Grove St. becomes 0301 Grove St. Another problem: The system doesn’t flag data entry mistakes, such as if a worker misidentified 301 Grove St. as 0031 Grove St. An employee giving a homeowner a tax exemption can cause the break to be revoked the next year by entering a single wrong digit on a different screen. It would take a complaint by the overcharged resident to bring the error to light.

It’s taking what seems like forever to drag Chu’s office into the present century. After a bidding process, San Francisco awarded contracts in November to Sapient Corp. and Carahsoft Technology Corp., which provides the Salesforce software licenses. The new system, slated for completion in 2022, needs to address problems that can’t be solved in the App Store. Although tech companies like to fancy themselves altruists and world changers, Chu says they “have not really focused on solving the mundane problems of basic core government functions.” Private developers have created an app that lets people report poop on San Francisco streets. (It’s called [8]Snapcrap. Yes, really.) Designing something to help the city pay for the cleanup? Different story.

Municipal offices across the country are struggling to do their jobs with obsolete gear that can often be expensive and time-consuming to replace

Police department
Issue: [9]The system for storing and tracking crime reports is more than 20 years old, doesn’t comply with the national incident reporting system, and can’t link to other databases.
Upgrade cost: $4 million

Office of emergency services
Issue: Twelve different [10]radio systems with portions dating to the 1970s prevent units from talking to one another.
Cost: $23 million

CourtsIssue: Case-tracking software to replace the Texas county’s outdated programs began development in 2012 but has faced repeated delays, leaving judges stuck with inefficient systems.

Cost: More than $30 million

Municipal courtIssue: A 1980s system for court records relies on typewriters. New software would ease the workload and reduce errors from having to repeatedly enter information.

Cost: $400,000

NJ Transit
Issue: The nation’s second-biggest [11]commuter railroad uses paper forms to procure its equipment and supplies. It wants a central, digital inventory.
Cost: Unknown

Streets departmentIssue: Control systems for most of the city’s 3,000 traffic signals date to the 1960s and make it harder for the department to manage congestion.

Cost (per intersection): $175,000 to $735,000

To contact the editor responsible for this story: Jeff Muskus at jmuskus@bloomberg.net, Flynn McRoberts

References

Visible links
1. https://sfgov.legistar.com/View.ashx?M=F&ID=6701050&GUID=2B1E9E29-E7B9-404E-9859-2F711A80F832
2. https://bloustein.rutgers.edu/pfeiffer/
3. https://s1.q4cdn.com/454432842/files/doc_financials/2019/q2/Q219-CRM-Earnings-Call-Transcript.PDF
4. http://www.washingtonpost.com/wp-dyn/content/article/2009/11/04/AR2009110402413.html?noredirect=on&sid=ST2009110403012
5. https://mn.gov/governor/news/#/detail/appId/1/id/371388
6. http://www.pti.org/about/ashark.asp
7. https://www.sfassessor.org/news/strength-property-tax-helps-sf-recognize-415-million-additional-funding
8. https://itunes.apple.com/us/app/snapcrap/id1436238261
9. https://www.baltimorepolice.org/sites/default/files/General%20Website%20PDFs/BPD_Final_Technology_Inventory_Study_06-21-18.pdf
10. https://gobroomecounty.co.broome.ny.us/node/58540
11. https://www.nj.gov/governor/docs/20181005NJTransitFinalReport.pdf

HackerNewsBot debug: Calculated post rank: 89 - Loop: 114 - Rank min: 80 - Author rank: 43

Does Carnitine from Red Meat Cause Heart Disease?

A detailed critique of Koeth and Hazen's study linking carnitine in red meat to heart disease. Their experiments fail to show a plausible connection.
Article word count: 3800

HN Discussion: https://news.ycombinator.com/item?id=19235684
Posted by sridca (karma: 1329)
Post stats: Points: 57 - Comments: 68 - 2019-02-23T21:51:45Z

\#HackerNews #carnitine #cause #disease #does #from #heart #meat #red

---

Article content:



[1]Meat heart licensedHold on to your hats — this is going to be a wild ride…

This post is a detailed critique of the following study:

[2]Koeth RA et al. Intestinal microbiota metabolism of L-carnitine, a nutrient in red meat, promotes atherosclerosis. Nature Medicine April 7 2013 [e-pub ahead of print].

The Hullabaloo over Hazen and Koeth’s Red Meat Study

A few days ago, a brand new study by Dr. Stanley Hazen’s group at the Cleveland Clinic was published, incriminating an unfamiliar ingredient within red meat as the cause of heart disease. The New York Times trumpeted:

[3]CULPRIT IN HEART DISEASE GOES BEYOND MEAT’S FAT

“The real culprit, they proposed, was a little-studied chemical that is burped out by bacteria in the intestines after people eat red meat. It is quickly converted by the liver into yet another little-studied chemical called TMAO that gets into the blood and increases the risk of heart disease.” (Gina Kolata, New York Times 4/7/13)

This article received a lot of attention, so I was asked by readers and friends to comment on it. This new study is actually a conglomeration of mouse experiments, human clinical studies, and human epidemiological studies, which the authors then weave together to make their case against red meat. Straightforward it is NOT. It is a many-headed beast, armed with tentacles and suckers, but is smart, elegant in many ways, and deserving of detailed scrutiny.

Much like a game of Twister, the authors stretch so far to try to connect the dots between their studies that their sophisticated theory ultimately collapses into a sad, lifeless heap. The bottom line is that there are no new reasons to fear red meat and no proof within this study (or any other study, for that matter) that red meat causes heart disease. But if you want to know how I came to this conclusion, trudge forth, gentle reader. And Godspeed.

The Authors’ Beliefs

(or why they conducted these studies in the first place)

The article opens this way (emphases mine):

“The high level of meat consumption in the developed world is LINKED to CVD (cardiovascular disease) risk, PRESUMABLY owing to the large content of saturated fats and cholesterol in meat. However, a recent meta-analysis of prospective cohort studies showed NO ASSOCIATION between dietary saturated fat intake and CVD, prompting the SUGGESTION that other environmental exposures LINKED to increased meat consumption are responsible.” [Koeth 2013]

If after more than 40 years of studying meat and heart disease, this is the strongest statement intelligent scientists can make condemning meat, then maybe, just maybe, the theory is weak…or…heaven forbid…completely false.

Allow me to summarize the authors’ line of reasoning:

1. Red meat must cause heart disease somehow, because [4]epidemiological studies (which have no power to demonstrate cause and effect) suggest that people who eat more red meat are at higher risk for heart disease. Epidemiological studies also suggest that vegetarians have lower rates of heart disease than omnivores [click [5]HERE to read why this may have nothing to do with meat.]
2. But a recent meta-analysis showed no connection between dietary saturated fat and heart disease, so fat is not the culprit after all…
3. We need to find a new villainous ingredient within red meat to blame for heart disease, because the possibility that red meat might NOT cause heart disease either hasn’t occurred to us, or sounds preposterous to us. [Perhaps they are not aware that some groups of [6]people eating very high-meat diets, like the Inuit, had very low rates of heart disease. I will never understand how intelligent people can believe that an ancient food is causing a modern disease.]
4. Hey…red meat contains more L-carnitine than white meat, and plant foods are extremely low in carnitine, so maybe L-carnitine causes heart disease!
5. Oh…but wait…L-carnitine is a vital nutrient in our bodies. L-carnitine is used to transport fat into the mitochondria of our cells so it can be burned for energy. It is so essential that if we don’t eat enough of it, we go out of our way to make it from scratch. Since L-carnitine itself is innocent, we have to work extra super hard to connect it to heart disease. How can we can pin the tail on this red-meat-filled donkey?

But first, a map of the forest, so we won’t get disoriented in the trees.

Introducing Carnitine

Carnis is the Latin word for meat. Because carnitine is vital to animal energy production, all animal foods contain carnitine, but red meat has a LOT more than other foods because dark muscle fibers rely most heavily on fat for energy:

Table 1: Selected food sources of carnitine

Food Milligrams (mg)
Beef steak, cooked, 4 ounces 56-162
Ground beef, cooked, 4 ounces 87-99
Milk, whole, 1 cup 8
Codfish, cooked, 4 ounces 4-7
Chicken breast, cooked, 4 ounces 3-5
Ice cream, ½ cup 3
Cheese, cheddar, 2 ounces 2
Whole-wheat bread, 2 slices 0.2
Asparagus, cooked, ½ cup 0.1

Source: [7]http://ods.od.nih.gov/factsheets/Carnitine-HealthProfessional/

Carnitine in meat exists as “acetyl-L-carnitine esters”, not as free carnitine. All this means is that meaty carnitine has special little attachments that make it easier for the small intestine to absorb (i.e. more “bioavailable”) than free L-carnitine. If we eat meat, we absorb a large percentage of the natural form of carnitine it contains. Vegans and vegetarians absorb more carnitine (66-86%) than meat-eaters (54-72%), because the body needs carnitine and it’s easier to get it from food than to make it from scratch. By contrast, if we swallow a supplement containing free L-carnitine, we only absorb about 10 to 20% of it.

Carnitine that is not absorbed by the small intestine can make it all the way down to the colon, where some types of bacteria can break it down, releasing a by-product called “TMA” (trimethylamine). TMA is a gas that smells like rotten fish. TMA wafts into the bloodstream and makes its way to the liver, where special enzymes convert it into TMAO (trimethylamine oxide)—an odorless substance that is easily excreted in the urine. TMAO is the molecule that Dr. Hazen’s group thinks causes heart disease.

Why would our bodies contain an enzyme that would go out of its way to deliberately turn a rotten gas (TMA) into a substance that can kill us (TMAO)? I doubt it would…While TMA itself does not seem to be toxic, it can combine with proteins to form potentially cancer-causing [8]nitrosamines. This may be just one reason why the liver works to transform it into something the body can get rid of.

Hey, what about CHOLINE?

Carnitine is not the only nutrient that bacteria can turn into TMA. Another common food molecule—choline—can also be turned into TMA. Choline is found in all kinds of foods, not just in meat:

[mg choline per 100g of food]
\* Egg yolks                                 680
\* Egg whites                               270
\* Liver                                        195 – 333
\* Toasted wheat germ                152
\* Meat/seafood                           34 – 103
\* Nuts                                         29 – 72
\* Brussels sprouts/broccoli         40
Source: [9]http://www.nal.usda.gov/fnic/foodcomp/Data/Choline/Choline.pdf

“The normal human diet contains approximately 500 mg of free choline/d and humans do not usually smell “fishy” unless they are treated with large oral doses of choline. Supplementary choline is ingested in “health food” preparations by many individuals, and choline supplements…”

[Zeisel]Something’s fishy

If TMAO is bad for us, we should also stop eating saltwater fish. We don’t even need bacteria to generate TMAO from fish—fish naturally contain TMAO; they use it to regulate their fluid balance so they won’t shrivel up in the salty waters of the sea:

Flounder: 400 mg/100 g

Alaskan Pollack and Cod: 1000 mg/100 g

[Sikorski 1990]

So, why pick on red meat, when TMAO can result from the digestion of other foods, such as egg whites, fish, wheat germ, nuts, and cruciferous vegetables?

The authors acknowledge that choline can form TMA and TMAO but do not mention that choline comes from plant foods as well as animal foods, and they do not mention fish TMAO at all. Either they did not do their homework, or their biases have blinded them to the facts. Such is human nature—believing is seeing.

Onward.

The Authors’ Arguments

(or how they think TMAO might cause heart attacks)

In order to draw the conclusion that red meat causes heart disease, you have to buy the following series of arguments. If you read only these 6 points, and you are someone who wants to believe that red meat is bad for you, I can see how you would come away feeling as if your beliefs are validated.
\* Point #1.  People with heart disease tend to have higher TMAO levels [Wang 2011].
\* Point #2.  Vegans and vegetarians naturally have lower TMAO levels.
\* Point #3.  Vegans and vegetarians produce less TMAO after consuming steak and L-carnitine than meat-eaters do.
\* Point #4.  Vegans and vegetarians have different kinds of bacteria living in their colon than omnivores do, and this may explain why they produce less TMAO.
\* Point #5.  L-carnitine supplements increase atherosclerosis in genetically-altered mice.
\* Point #6.   TMAO interferes with “Reverse Cholesterol Transportt” (RCT) in genetically-altered mice, so it’s harder for their bodies to get rid of excess cholesterol.
ONE GIANT LEAP FOR MANKIND

If you eat red meat, your body will fill up with cholesterol and you will have a heart attack.

Counterpoint #1

The observation that people with heart disease are more likely to have higher TMAO levels does not mean that TMAO causes heart attacks (and the authors do not claim that it does). It might or it might not. TMAO could simply be an innocent bystander—a red (meat) herring. We have no idea why people who have heart disease tend to have higher TMAO levels. Most importantly, researchers did not ask these people what they eat. Did they eat healthy whole foods diets or junky diets? How many of them were vegans? Vegetarians? Omnivores? Were the ones with the highest TMAO levels all omnivores? Were the ones with the lowest TMAO levels vegans? That would have been very interesting and helpful information.

Counterpoint #2

Not all vegans and vegetarians have lower TMAO levels compared to omnivores. From this figure, taken from Hazen’s study, it looks like only a small percentage of them do:

[10]carnitine vegan omnivore figure

[The lowest horizontal line = 10^th percentile. The bottom of each box = 25^th percentile. The line inside each box = 50^th percentile. The top of each box = 75^th percentile. The highest horizontal line = 90^th percentile].

The fact that there is so much overlap in TMAO levels between meat-eaters and non-meat-eaters suggests that there is something else about diet–something other than the presence or absence of meat– that is playing a significant role in TMAO levels. Dietary choline is the most logical and likely explanation, but may not be the only one.

Counterpoint #3

a. The form of carnitine found in red meat—acetyl-L-carnitine— was not used in the study; free L-carnitine supplements were used in the study instead. Because these supplements contain the form of carnitine that is hardest to absorb, the majority of it bypasses the small intestines and makes it down to the colon, where bacteria can turn it into the maximum amount of TMA (and TMAO) possible. This study does not tell us whether acetyl-L-carnitine—the form of carnitine found in red meat— raises TMAO levels. Even if TMAO turns out to be the scourge of the West, all you will have shown with this study is that people who want to avoid heart attacks should not take L-carnitine supplements with their steak.

b. NONE of the subjects was fed steak alone—they all received either L-carnitine supplements alone or in combination with steak. There is no proof anywhere in this paper that simply eating steak all by itself will raise anyone’s TMAO levels. In order to convince people that steak raises TMAO levels, you have to include a steak-only experiment. Any high school science student could tell you that.

Here’s an example of what the scientists did:

Experiment: Give one male vegan and one female omnivore each an 8-oz steak (which contains 180 mg of acetyl L-carnitine) plus a capsule containing 250 mg of free L-carnitine and watch what happens to their TMAO levels. [Poor vegan man–he really took one for the team. Give that man a year’s supply of tofu!]

Result: The vegan man produced virtually no TMAO after the steak + carnitine supplement whereas the meat-eating woman generated a TMAO spike in her bloodstream.

Given that studies have shown vegans absorb more carnitine than meat-eaters, isn’t it possible that the vegan man, whose body may have been pining for some long-overdue, prefabricated carnitine, absorbed much more carnitine than the omnivorous woman, so that much less of it made it to his colon to be turned into TMA? Isn’t it possible that the vegan man didn’t generate a TMAO spike because his liver for some reason couldn’t turn TMA into TMAO? In this case, he would have generated a TMA spike, but TMA was not measured. Chris Masterjohn wrote an [11]excellent review of this study and proposes other interesting possibilities, including gender differences and vitamin deficiency issues. He also does a beautiful job of analyzing the omnivore vs vegan/vegetarian experiments.

Counterpoint #4

Yes, what we eat determines the types of bacteria in our colon, but we have no idea which mix of bacteria is healthiest for us, we have no idea which diet encourages the best mix of bacteria, and we have no idea whether any of this has anything to do with heart disease.

The researchers detailed the interesting differences in bacterial patterns they discovered between meat-eaters and non-meat-eaters. One type of intestinal bacteria called Prevotella was strongly associated with higher TMAO production. Four of the human volunteers were found to have bacterial colonies in their colons that were rich in Prevotella. Were all four of these individuals meat-eaters? No. Three of them were omnivores (5.9% of all omnivores in the study), and one was not (3.8% of all non-meat-eaters). To quote the authors, this suggested:

“more complexity in the human gut microbiome than anticipated…Indeed, other studies have demonstrated variable results in associating human bacterial genera…including Prevotella, to omnivorous and vegetarian eating habits.”

Translation: Even if you are convinced that having Prevotella colonies setting up house in your innards is bad for you, eating a vegan/vegetarian diet will not guarantee you a low-Prevotella colon.

Counterpoint #5
\* Yes, L-carnitine supplements caused an increase in atherosclerosis in genetically altered mice. However, L-carnitine is not red meat.  It is not even the form of carnitine found in red meat (acetyl-L-carnitine).
\* The dose of L-carnitine used in these mice was extremely high. According to [12]Chris Masterjohn, it was the equivalent of a human eating 1000 sirloin steaks.
\* For reasons I don’t understand, the mice used in these studies were genetically-altered so that they were missing a gene (apoE) required for normal cholesterol processing.  These mice are popular with scientists who study heart disease because they are very good at developing atherosclerosis.  It’s already a big stretch to apply information from mouse studies to human health, why widen the gap by using unnaturally defective mice?  Absurd.
Counterpoint #6

Now here’s where the authors really jumped the shark. They fed their miscreant mice standard mouse chow supplemented with L-carnitine, choline, or TMAO itself. Then a trusty lab assistant was charged with the delightful task of measuring how much cholesterol the mice…um…released. Makes me nostalgic for the 7 years I spent as a lab tech…not. They found that mouse pellets from animals fed TMAO contained 30% less cholesterol than those fed unsupplemented chow. This is how they propose TMAO causes human heart disease. There is so much I could say about these studies, but the take-home message for your refrigerator magnet is this:

Just because the cage droppings of mutant mice deliberately fed TMAO contained less cholesterol than those who were not fed TMAO does not mean that a human who eats a steak is going to have a heart attack.

My BIGGEST BEEF with this study

Like most studies that try to connect red meat to human disease, this collection of studies does not control for refined carbohydrate intake. We do not know how healthy the diets of the human volunteers were before this study began. Were the vegans and vegetarians health-conscious types who avoided junk foods? What about the omnivores? What if it turned out that everyone who had a higher TMAO level (including the veg/vegans with higher TMAO) also happened to eat a lot more sugar and flour?

After all, if there is one clear culprit emerging from the piles of studies about diet and heart disease, it is refined carbohydrate—not saturated fat, not cholesterol, not red meat. So, in this day and age, in my humble opinion, if you are going to conduct a study of diet and heart disease, you simply must take refined carbohydrate into consideration in order to be taken seriously.

Let’s look at two clear examples in this study of how refined carbohydrate intake was completely ignored by the authors.

The authors measured TMAO levels in 2,595 people and concluded that there was a dose-dependent relationship between TMAO levels and cardiovascular disease (CVD) risk, after adjustment for “traditional CVD risk factors”, which were age, gender, diabetes, smoking, blood pressure, cholesterol levels, and medications. Notice the absence of refined carbohydrate intake, fasting blood glucose, hemoglobin A1C or any other related values.

But hope was kindled when, within a description of one of their mouse studies, I came across this promising line:

“Of note, the increase in atherosclerotic plaque burden with dietary L-carnitine occurred in the absence of proatherogenic changes in plasma lipid, lipoprotein, glucose, or insulin levels…”

Excellent–they acknowledged glucose and insulin as potential risk factors! Had they actually determined that carnitine caused atherosclerosis despite healthy glucose and insulin levels? The sentence above, which is found in the primary article, would lead you to believe this, but if you dig through the 40-page supplementary information, you unearth this telling table:

[13]carnitine mouse values best

As you can see, the triglyceride, cholesterol, glucose and insulin levels of mice being fed normal chow are already pretty high (insulin levels at baseline convert to 16.3 microunits/ml). When carnitine is added, there is no statistically significant change in any of these levels (as indicated by P values greater than 0.05). The authors phrased their sentence carefully so that what they say is true—there were no proatherogenic (artery-clogging) changes in these profiles. They were already proatherogenic to begin with.

These mice, like most laboratory animals, are fed unbelievably junky diets. In this case, they were eating a “standard chow control diet”, called [14]Teklad 2018:

Ingredients (in descending order of inclusion)- Ground wheat, ground corn, wheat middlings, dehulled soybean meal, corn gluten meal, soybean oil, calcium carbonate, dicalcium phosphate, brewers dried yeast, iodized salt, L-lysine, DL-methionine, choline chloride, kaolin, magnesium oxide, vitamin E acetate, menadione sodium bisulfite complex (source of vitamin K activity), manganous oxide, ferrous sulfate, zinc oxide, niacin, calcium pantothenate, copper sulfate, pyridoxine hydrochloride, riboflavin, thiamin mononitrate, vitamin A acetate, calcium iodate, vitamin B12 supplement, folic acid, biotin, vitamin D3 supplement, cobalt carbonate.

This chow is comprised almost entirely of processed wheat, corn, and soy—nearly 100% refined carbohydrate, held together with soybean oil. This is not a diet that exists in nature. Poor little mice. No wonder manufacturers have to supplement this stuff with 18 vitamins and minerals. Look familiar? Is anyone reminded of the ingredient list on the side of your typical cereal box? Coat these mouse pellets with chocolate frosting or sprinkles and you’ve got yourself a yummy breakfast treat 🙂

The authors provide evidence that vegans/vegetarians have a different mix of bacteria living in their colon than omnivores, and they do an excellent job of convincing us that you can’t generate TMAO without bacteria (humans do not possess this capability). Then, because they believe that red meat is bad for us, they try to connect red meat to bacterial metabolism of carnitine to TMAO. Now allow me to do the same thing with refined carbs. It took me about 3 minutes to find this reference:

“Phosphotransferase systems involved in microbial processing of carbohydrates also were found to be over expressed in the intestines of obese people, notably from Prevotellaceae…”

[Kallus]I am not saying that carbs increase Prevotella activity or that Prevotella causes heart attacks. All I’m saying is that refined carbohydrates may also play an important role in what kind of bacteria elbow their way to your colonic buffet table.

GUILT BY ASSOCIATION

Based on the thousands of scientific articles I have read about food and health, in combination with my personal experience, clinical experience, and common sense, I find nothing to suggest that red meat is bad for people, and plenty of evidence that red meat is good for people (see my [15]meat page). Every article I’ve ever read that tries to blame meat for our health problems fails to to take refined carbohydrate into consideration. In my opinion, damning nutritious meat for your heart attack is like pouring sugar into your gas tank and then cursing the gasoline when your car breaks down.

If you want to scare me away from eating meat using mouse experiments, you are going to have to raise a slew of mice on a healthy whole foods vegetarian diet, prove to me that those mice show no signs of cardiovascular disease, then start feeding them one miniature sirloin steak for dinner every night until they all clutch their furry little chests, keel over, and die.

HAS THE FAT LADY SUNG?

As far as I can tell, the authors’ theory that red meat provides carnitine for bacteria to transform into TMA which our liver then converts to TMAO, which causes our macrophages to fill up with cholesterol, block our arteries, and cause heart attacks is just that–a theory–full of sound and fury, signifying nothing.

To read my critique of the World Health Organization’s 2015 proclamation that red and processed meats cause cancer: [16]WHO Says Meat Causes Cancer?

REFERENCES

Bennett BJ et al. Trimethylamine-N-Oxide, a Metabolite Associated with Atherosclerosis, Exhibits Complex Genetic and Dietary Regulation. Cell Metabolism 2013; 17: 49–60.

Kallus SJ and Brandt LJ. The Intestinal Microbiota and Obesity. J Clin Gastroenterol 2012;46:16–24.

Koeth RA et al. Intestinal microbiota metabolism of l-carnitine, a nutrient in red meat, promotes atherosclerosis. Nat Med Apr 7 2013 [Epub ahead of print].

Rebouche CJ. Kinetics, pharmacokinetics, and regulation of L-Carnitine and Acetyl-L-carnitine metabolism. Ann NY Acad Sci 2004; 1033: 30–41.

Reuter SE and Evans AM. Carnitine and Acylcarnitines: Pharmacokinetic, Pharmacological and Clinical Aspects. Clin Pharmacokinet 2012; 51(9): 553-572.

Sikorski ZE. Seafood: Resources, Nutritional Composition, and Preservation. 1990 CRC Press.

Wang Z et al. Gut flora metabolism of phosphatidylcholine promotes cardiovascular disease. Nature 2011; 472 (7341): 57-63.

Zeisel SH, et al. Conversion of Dietary Choline to Trimethylamine and Dimethylamine in Rats: Dose-Response Relationship. J Nutrition 1989; 119 (5): 800-4.

References

Visible links
1. http://diagnosisdiet.com/wp-content/uploads/2013/04/Meat-heart-licensed.jpg
2. http://www.ncbi.nlm.nih.gov/pubmed/23563705
3. http://www.nytimes.com/2013/04/08/health/study-points-to-new-culprit-in-heart-disease.html?_r=0
4. http://diagnosisdiet.com/epidemilogical-studies/
5. http://diagnosisdiet.com/diet/vegetarian-diets/
6. http://diagnosisdiet.com/all-meat-diets/
7. http://ods.od.nih.gov/factsheets/Carnitine-HealthProfessional/
8. http://diagnosisdiet.com/food/meats/
9. http://www.nal.usda.gov/fnic/foodcomp/Data/Choline/Choline.pdf
10. http://diagnosisdiet.com/wp-content/uploads/2013/04/carnitine-vegan-omnivore-figure.jpg
11. http://blog.cholesterol-and-health.com/
12. http://blog.cholesterol-and-health.com/
13. http://diagnosisdiet.com/wp-content/uploads/2013/04/carnitine-mouse-values-best.jpg
14. http://www.harlan.com/online_literature/teklad_lab_animal_diets
15. http://diagnosisdiet.com/food/meats/
16. http://www.diagnosisdiet.com/meat-and-cancer/

HackerNewsBot debug: Calculated post rank: 60 - Loop: 308 - Rank min: 60 - Author rank: 20

Open Letter from New York State Budget Director Robert Mujica Regarding Amazon

An open letter from New York State Budget Director Robert Mujica regarding Amazon.
Article word count: 1590

HN Discussion: https://news.ycombinator.com/item?id=19234877
Posted by agreen (karma: 229)
Post stats: Points: 134 - Comments: 124 - 2019-02-23T18:56:19Z

\#HackerNews #amazon #budget #director #from #letter #mujica #new #open #regarding #robert #state #york

---

Article content:



"As just about everyone in this state, if not the country, knows by now, Amazon has terminated its plans to bring its second headquarters to New York State. It is a tremendous loss for New Yorkers and I hope that at a minimum, we understand the lessons learned.

"In my 23 years in the State Capitol, three as Budget Director, Amazon was the single greatest economic development opportunity we have had. Amazon chose New York and Virginia after a year-long national competition with 234 cities and states vying for the 25,000-40,000 jobs. For a sense of scale, the next largest economic development project the state has completed was for approximately 1,000 jobs. People have been asking me for the past week what killed the Amazon deal. There were several factors.

"First, some labor unions attempted to exploit Amazonʼs New York entry. The RWDSU Union was interested in organizing the Whole Foods grocery store workers, a subsidiary owned by Amazon, and they deployed several ʼcommunity based organizationsʼ (which RWDSU funds) to oppose the Amazon transaction as negotiation leverage. It backfired. Initially, Whole Foods grocery stores had nothing to do with this transaction. It is a separate company. While Amazon is not a unionized workforce, Amazon had agreed to union construction and service worker jobs that would have provided 11,000 thousand union positions.

"New York State also has the most pro-worker legal protections of any state in the country. Organizing Amazon, or Whole Foods workers, or any company for that matter, is better pursued by allowing them to locate here and then making an effort to unionize the workers, rather than making unionization a bar to entrance. If New York only allows unionized companies to enter, our economy is unsustainable, and if one union becomes the enemy of other unions, the entire union movement - already in decline - is undermined and damaged.

"Second, some Queens politicians catered to minor, but vocal local political forces in opposition to the Amazon government incentives as ʼcorporate welfare.ʼ Ironically, much of the visible ʼlocalʼ opposition, which was happy to appear at press conferences and protest at City Council hearings during work hours, were actual organizers paid by one union: RWDSU. (If you are wondering if that is even legal, probably not). Even more ironic is these same elected officials all signed a letter of support for Amazon at the Long Island City location and in support of the application. They were all for it before Twitter convinced them to be against it.

"While there is always localized opposition, in this case it was taken to a new level. The State Senate transferred decision-making authority to a local Senator, who, after first supporting the Amazon project, is now vociferously opposed to it, and even recommended appointing him to a State panel charged with approving the projectʼs financing. Amazon assumed that the hostile appointment doomed the project. Of course the Governor would never accept a Senate nomination of an opponent to the project and the Governor told that to Amazon directly. The relevant question for Amazon then became whether the Senate would appoint an alternative who would approve the project.

"As newspapers have reported, Amazon called the Senate Leader and asked if she would appoint an alternative appointee who would support the project. The Senate would not commit to an alternative appointee supporting Amazon. That was the death knell. No rational company, or person for that matter, would assume the Senate would flip flop from appointing a staunch opponent of the project to appointing a supporter of the project. It defies logic. However, if that was their plan, Amazon needed a direct representation to that effect from the Senate. It never came. Indeed, to this day, the Senate has never said they would appoint a member who would support the project. Companies assume rational, logical behavior and cannot spend months and millions of dollars on approvals if ultimately the road is a dead end.

"Furthermore, opposing Amazon was not even good politics, as the politicians have learned since Amazon pulled out. They are like the dog that caught the car. They are now desperately and incredibly trying to explain their actions. They cannot. They are trying to justify their flip-flopping on the issue with false accusations that it was a ʼbackroom deal.ʼ Letʼs remember that as a condition of the competition, every bid was sealed to prevent governments from altering their bids to be more competitive. Empire State Development supported the numerous local applications in the state who wanted to bid for HQ2, but on the condition that the local elected officials and community supported it, and Long Island City was no exception.

"In working with New York City, we advanced Long Island Cityʼs application with the signed support of the areaʼs local elected officials, including State Senator Mike Gianaris and New York City Councilman Jimmy Van Bramer. Both Gianaris and Van Bramer flip-flopped on this position after Long Island City was chosen, distorting the facts of the agreement and mischaracterizing the tax subsidies as ʼa cash giveaway.ʼ Now that Amazon has pulled out, local politicians are feeling the backlash from the projectʼs previously silent supporters and are dissembling. Local senatorsʼ claims that their phone calls were not returned are particularly offensive, given that the local senator was the first person ESD President and CEO Howard Zemsky met with when we made the HQ2 announcement. I also remained in contact with him about the project as the State Budget Director, and he refused to sit on the community engagement board or even meet with Amazon representatives. Efforts were made to address legitimate concerns, all of which were ignored.

"Third, in retrospect, the State and the City could have done more to communicate the facts of the project and more aggressively correct the distortions. We assumed the benefits to be evident: 25,000-40,000 jobs located in a part of Queens that has not seen any significant commercial development in decades and a giant step forward in the tech sector, further diversifying our economy away from Wall Street and Real Estate. The polls showing seventy percent of New Yorkers supported Amazon provided false comfort that the political process would act responsibly and on behalf of all of their constituents, not just the vocal minority. We underestimated the effect of the oppositionʼs distortions and overestimated the intelligence and integrity of local elected officials.

"Incredibly, I have heard city and state elected officials who were opponents of the project claim that Amazon was getting $3 billion in government subsidies that could have been better spent on housing or transportation. This is either a blatant untruth or fundamental ignorance of basic math by a group of elected officials. The city and state ʼgaveʼ Amazon nothing. Amazon was to build their headquarters with union jobs and pay the city and state $27 billion in revenues. The city, through existing as-of-right tax credits, and the state through Excelsior Tax credits - a program approved by the same legislators railing against it - would provide up to $3 billion in tax relief, IF Amazon created the 25,000-40,000 jobs and thus generated $27 billion in revenue. You donʼt need to be the Stateʼs Budget Director to know that a nine to one return on your investment is a winner.

"The seventy percent of New Yorkers who supported Amazon and now vent their anger also bear responsibility and must learn that the silent majority should not be silent because they can lose to the vocalminority and self-interested politicians.

"It was wrong to manage this issue as if it were a single legislatorʼs political prerogative on a local matter. This was not a traffic signal or local zoning issue. Losing the Amazon project was not just a blow to Queens County, it hurt the whole State from Long Island to the Capitol Regionʼs nanotechnology corridor to the emerging Panasonic plant in Buffalo, and it was a bad reflection on every single local elected official. Legislators must realize there is a difference between playing politics and responsibly governing.

"Progressive politics and policies have been the signature of Governor Cuomoʼs administration. No state in the nation has more progressive accomplishments, and being the most progressive state in the nation means having the most stringent and aggressive protections and policies in place. We are proud that our values create a stronger, healthier, fairer work environment, but we shouldnʼt kid ourselves about how they impact our competitiveness when businesses consider where to locate. We are also proud of the unprecedented investments we make in education, healthcare, infrastructure and housing, but in order to fund them, we need a sustained tax base.

"As the political debate rages in this country, the Governor reminds us of the fact that ʼto be a progressive, there must be progress.ʼ The creation of opportunity and jobs is the engine that pulls the train and, as he also often says, ʼthe best social program is still a job.ʼ Without a tax base we are not financially able to achieve the laudable goals we seek.

"Make no mistake, at the end of the day we lost $27 billion, 25,000-40,000 jobs and a blow to our reputation of being ʼopen for business.ʼ The union that opposed the project gained nothing and cost other union members 11,000 good, high-paying jobs. The local politicians that catered to the hyper-political opposition hurt their own government colleagues and the economic interest of every constituent in their district. The true local residents who actually supported the project and its benefits for their community are badly hurt. Nothing was gained and much was lost. This should never happen again."

HackerNewsBot debug: Calculated post rank: 130 - Loop: 169 - Rank min: 100 - Author rank: 99

Facebook collects a wide range of private data from developers

Millions of smartphone users confess their most intimate secrets to apps, including personal health information. Unbeknown to most people, in many cases that data is being shared with someone else:…
Article word count: 63

HN Discussion: https://news.ycombinator.com/item?id=19226480
Posted by mudil (karma: 5384)
Post stats: Points: 148 - Comments: 53 - 2019-02-22T16:12:59Z

\#HackerNews #collects #data #developers #facebook #from #private #range #wide

---

Article content:




Millions of smartphone users confess their most intimate secrets to apps, including when they want to work on their belly fat or the price of the house they checked out last weekend. Other apps know users’ body weight, blood pressure, menstrual cycles or pregnancy status.

Unbeknown to most people, in many cases that data is being shared with someone else: Facebook Inc.

The...

HackerNewsBot debug: Calculated post rank: 116 - Loop: 308 - Rank min: 100 - Author rank: 57

Pee, Not Chlorine, Causes Red Eyes from Swimming Pools: CDC (2015)

U.S. public health officials confirm that it's not chlorine giving red-eye to swimmers, as many believe — it's people who pee in the pool.
Article word count: 665

HN Discussion: https://news.ycombinator.com/item?id=19208700
Posted by colinprince (karma: 15805)
Post stats: Points: 134 - Comments: 67 - 2019-02-20T15:41:54Z

\#HackerNews #2015 #causes #cdc #chlorine #eyes #from #not #pee #pools #red #swimming

---

Article content:



Chlorine has long had a bad rap for irritating the eyes of swimmers, especially in crowded public pools.

As it turns out, however, itʼs not the chemical itself turning your eyes red after a swim — itʼs everything else in the water that chlorine goes in to kill.

Specifically, human urine.

The U.S. Centers for Disease Control (CDC) recently teamed up with both the Water Quality and Health Council and the National Swimming Pool Foundation to warn the public about health risks associated with summer.

"Smell that ʼchlorineʼ?" reads [1]a fact sheet released by the coalition last month. "Itʼs not what you think. What you smell are actually chemicals that form when chlorine mixes with pee, poop, sweat, and dirt from swimmersʼ bodies.… These chemicals — not chlorine — can cause your eyes to get red and sting, make your nose run, and make you cough."

Swimming with someone who has diarrhea is more than just gross. According to the CDC, it can be dangerous. (U.S. Centers for Disease Control)

A [2]companion report published by the National Swimming Pool Foundation, an American non-profit dedicated to aquatic health and safety, elaborates on what it is about urine that leads to red eyes.

"Chlorine and other disinfectants are added to a swimming pool to destroy germs," said Michele Hlavsa, chief of CDCʼs Healthy Swimming Program, in a release. "Peeing in a pool depletes chlorine and actually produces an irritant that makes peopleʼs eyes turn red."

To eliminate the irritants caused by nitrogen-containing compounds found in urine, more chlorine may need to be added to a pool, she said.

Red eyes are one of several colour-related topics tackled by the nationwide campaign.

Another relates to what the swimming pool foundation calls "the most common pool myth of all time" — one that nearly half of all Americans surveyed by researchers believed was true.

"Parents have long used the story of a chemical that changes colour in the presence of pee to keep their children from peeing in the pool," reads the report.

Foundation CEO Thomas M. Lachocki made the truth clear, saying "there isnʼt a dye that turns red. Itʼs the eyes that turn red. Swimmersʼ eyes are the real colour indicator that someone might have peed in a pool."
People who get into the water can carry in and spread germs. <a href="https://twitter.com/hashtag/SwimHealthy?src=hash">#SwimHealthy</a> <a href="http://t.co/wsCJw3zWB0">http://t.co/wsCJw3zWB0</a> <a href="http://t.co/4A2liL48Zm">pic.twitter.com/4A2liL48Zm</a>

—[3]@CDCgov
As gross as that sounds, itʼs one of the mildest potential ailments that can be caused by human waste in a public pool.

Swimming at an indoor pool is particularly risky, [4]according to the CDC, as the irritants mentioned above can move into the air surrounding a pool and trigger coughing, wheezing, or even asthma attacks.

And then thereʼs the issue of infectious diseases.

In an interview about the campaign with Womenʼs Health, the CDC Healthy Water programʼs associate director noted that thereʼs [5]been an increase in outbreaks of recreational water illnesses over the past decade.

Cryptosporidium is a leading cause of waterborne disease, according to the U.S. Centers For Disease Control. (U.S. Centers for Disease Control)

The reason for this? People who swim [6]while they have diarrhea and unleash even very tiny amounts of germs like Cryptosporidium (or "crypto" for short), norovirus, and E. coli. into the water.

"Diarrhea and swimming donʼt mix!" reads [7]the CDCʼs website. "Swimmers who are sick with diarrhea — or who have been sick in the last two weeks — risk contaminating pool water with germs. Certain germs that cause diarrhea can live from minutes to days in pools, even if the pool is well-maintained. Once the pool has been contaminated, all it takes is for someone to swallow a small amount of pool water to become infected."

Approximately 58 per cent of Canadians admitted [8]to peeing in the pool at least once in a recent survey of 9,500 people conducted by Travelocity.

If youʼre one of them, Hlavsa has some advice:

"The solution isnʼt rocket science; itʼs common courtesy. Swimmers should use the pool to swim, the restroom to pee and the showers to wash up before getting in the pool. Itʼs that simple."

References

Visible links
1. http://www.cdc.gov/healthywater/pdf/swimming/resources/share-fun-not-germs-508c.pdf
2. http://nspf.org/en/NewsNew/15-05-19/Healthy_Pools_Red_Eye.aspx
3. https://twitter.com/CDCgov/status/601024696486731776
4. http://www.cdc.gov/healthywater/swimming/pools/irritants-indoor-pool-air-quality.html
5. http://www.womenshealthmag.com/health/pee-in-pool-water
6. http://www.cdc.gov/healthywater/swimming/rwi/
7. http://www.cdc.gov/healthywater/swimming/rwi/illnesses/diarrhea-swimming.html
8. http://www.huffingtonpost.com/travelzoo/64-of-americans-pee-in-po_b_7598626.html

HackerNewsBot debug: Calculated post rank: 111 - Loop: 102 - Rank min: 100 - Author rank: 82

Could 'Oumuamua be an icy fractal aggregate ejected from a protoplanetary disk?

Not to put too fine a point on it, but what the frak is 'Oumuamua?
Article word count: 1356

HN Discussion: https://news.ycombinator.com/item?id=19201582
Posted by Santosh83 (karma: 2214)
Post stats: Points: 309 - Comments: 151 - 2019-02-19T18:45:57Z

\#HackerNews #aggregate #could #disk #ejected #fractal #from #icy #oumuamua #protoplanetary

---

Article content:




Not to put too fine a point on it, but what the frak is ʼOumuamua?

Oh, you remember ʼOumuamua. It caused quite a stir last year; [1]first seen in late 2017 by the Pan-STARRS survey telescope in Hawaii, it was quickly found to have a very unusual orbit. Instead of the usual ellipse or circle around the Sun like normal solar system objects, it was found to have a hyperbolic orbit. That means [2]it was moving too quickly to be bound to the Sun, and that, in turn, means it came from Out There. Like really out there: interstellar space, the void between the stars.

Subsequent observations confirmed it: ʼOumuamua was just passing through the solar system, with so much extra velocity (about 25 km/sec) that it was moving faster than the Sunʼs escape velocity. This was a one-time visitor, screaming through the solar system and heading back out into The Black once again.

IFrame: [3]media-youtube-q4etuh1o7ac
Video of A2017U1 animation

That was certainly enough to make it the object of intense scrutiny. Weʼd never seen something from interstellar space pass through the solar system before! But what was it? At first it was classified as a comet, then an asteroid, and [4]then maybe a comet again (this confusion is reflected in its provisional designations; at first it was A/2017 U1, for "asteroid", then C/2017 U1, for "comet", then finally I/2017 U1, for "interstellar"). It was hard to tell what it was; it was too small, faint, and far away to get good observations, and worse, it was only seen on its way out, so it was farther from us literally every day.

Then another very weird thing happened: More observations allowed a better determination of its trajectory, and [5]it was found that it wasnʼt slowing down fast enough. As it moves away, the Sunʼs gravity pulls on it, slowing it down … but it wasnʼt slowing down enough.

Some force was acting on it, accelerating it very slightly. Comets are made of rock and ice, so maybe the ice was turning into gas, and as this was blown off it acted like a very gentle rocket. The problem with this is that no such venting was detected. If it were like comets in our solar system, youʼd expect to see lots of carbon monoxide (CO) and carbon dioxide (CO2) coming from it, but none was seen. So maybe it was some other kind of ice, like water. But again, if it is like our local comets, it would take so much water that weʼd have noticed.

Thatʼs when a couple of astronomers posited something interesting: Maybe this force was radiation pressure, literally the force of sunlight hitting it and giving it a tiny push. That makes some sense, but for the math to work out with the acceleration seen, ʼOumuamua had to be flat. Like, really flat: So thin that it looked more like a solar sail, a very thin sheet of material designed to catch sunlight and accelerate. But that, in turn, meant that ʼOumuamua was artificial. As in, a spaceship.

Besides the obvious (it seems like a big leap!), [6]I have my problems with this idea. Not much has changed with that hypothesis since I wrote that, and while I wouldnʼt dismiss it being an alien probe out of hand, the evidence doesnʼt support that conclusion, and in fact points against it.

So, I ask again: What the frak is ʼOumuamua?

[7]A new paper has come out that might have a solution, and itʼs really clever. Maybe ʼOumuamuaʼs not flat. Maybe itʼs fluffy.

When the astronomers speculated it might be a thin and flat, giving it a large area like a sail, [8]they had to assume a density for it. Thatʼs because the amount of pressure sunlight exerts is very small, so if an object is massive it has to be spread out very thin and big to catch enough sunlight to accelerate it enough to match the observations. So they assumed it had some normal density like 1 – 3 grams per cubic centimeter (roughly somewhere between the density of water to rock).

The new paper turns that around. Instead of assuming a density to find the area, letʼs assume the size determined [9]using normal methods is correct, use that to get an area, and from there get the density needed to match the observations.

Assuming a size for ʼOumuamua of 50 – 130 meters, what they get is a very low density: About 0.00005 grams per cc. Thatʼs incredibly low, and at first it seems ridiculously so. Thatʼs 100 times less dense than air! No solid object could have a density that low!

… so what if itʼs not solid?

Iʼm not saying itʼs hollow, but maybe itʼs really porous. Like Styrofoam or Swiss cheese, but much much holey-er. Is there anything natural like that?

It turns out, surprisingly, yes. When stars are very young, [10]they have a huge disk of material swirling around them; itʼs from this material that planets form. Out far from the star, where temperatures in the disk are cold, teeny tiny grains of dust and water ice [11]can stick together in funny shapes, creating fractals. These are branching structures that arenʼt regularly spaced like crystals; instead they have a structure thatʼs called self-similar: It looks the same no matter how much you zoom in or out. I know that sounds weird, but [12]this video will help:

IFrame: [13]media-youtube-wfttdf3i6ug
Video of What Is A Fractal (and what are they good for)?

We see this a lot in nature, including in snowflakes. Materials made in a fractal pattern can be very porous, and in fact out in that protoplanetary disk around a young star, physical models show that objects can grow fractally until theyʼre as big as ‘Oumuamua, and have those extremely low densities needed to account for its weird behavior.

So ʼOumuamua doesnʼt have to be a spaceship. It just has to be a snowflake! A three-dimensionally constructed phenomenally porous low-density snowflake.

And in fact this actually makes sense. The fact we see something like ʼOumuamua coming from interstellar space means we can try to estimate how many of them are out there at any time. Making some simple assumptions, the numbers you get [14]are way too high to be explained by having the galaxy filled with such beasts. It seems far more likely that objects like ʼOumuamua are relatively rare, and that means it likely came from someplace close by (if it came from farther away, the odds are even lower weʼd ever see one). Working out the math, the new paper suggests it came from a nearby star, and one thatʼs relatively young (less than 100 million years). It formed out in the disk, and got ejected somehow, likely from a planet forming nearby giving it a boost from its gravity.

I have to say, I love this. It may not be as sexy or headline-grabbing as saying itʼs an alien probe, but itʼs very clever, it uses known materials and math, and the conclusion even fits in with what we know about how planets form. That makes it pretty dang cool in its own right. If I had to bet, Iʼd put a lot more money on this idea being correct over ʼOumuamua being an artifact from another civilization.

It may be weʼll never know; itʼs moving pretty rapidly relative to us, and is very far away now — itʼs currently over 1.6 billion kilometers away. [15]In a year itʼll be 900 million km further. So itʼs not like we can catch up to it anytime soon.

Weʼll just have to wait for another such object … which, according to this new idea, may take a long time. But weʼre scanning the skies with better tech, seeing deeper, seeing fainter objects, all the time now. I certainly hope we find more beasties like this one. They can tell us so much about how planets form in other star systems, which is pretty hard to figure out from dozens or hundreds of light years away. Itʼs a lot easier when they obligingly send bits of their building materials to us.

References

Visible links
1. https://www.syfy.com/syfywire/breaking-have-astronomers-discovered-our-first-interstellar-visitor-note-not-aliens
2. https://www.syfy.com/syfywire/breaking-have-astronomers-discovered-our-first-interstellar-visitor-note-not-aliens
3. https://www.youtube.com/embed/q4EtUH1O7ac?wmode=opaque&controls=&enablejsapi=1&playerapiid=media-youtube-q4etuh1o7ac
4. https://www.syfy.com/syfywire/so-oumuamua-is-likely-a-comet-after-all
5. https://www.syfy.com/syfywire/so-oumuamua-is-likely-a-comet-after-all
6. https://www.syfy.com/syfywire/is-oumuamua-an-interstellar-spaceship-im-still-going-with-no
7. https://arxiv.org/pdf/1902.04100.pdf
8. https://arxiv.org/pdf/1810.11490.pdf
9. https://www.syfy.com/syfywire/basteroid
10. https://www.syfy.com/syfywire/hey-we-were-right-planets-found-around-the-baby-star-hd-163296
11. https://arxiv.org/pdf/1402.7132.pdf
12.
13. https://www.youtube.com/embed/WFtTdf3I6Ug?wmode=opaque&controls=&enablejsapi=1&playerapiid=media-youtube-wfttdf3i6ug
14. https://arxiv.org/pdf/1810.02148.pdf
15. https://ssd.jpl.nasa.gov/sbdb.cgi?sstr=%27Oumuamua;old=0;orb=1;cov=0;log=0;cad=0#orb

HackerNewsBot debug: Calculated post rank: 256 - Loop: 177 - Rank min: 100 - Author rank: 42

Drug Combo Creates New Neurons from Neighboring Cells

A simple drug cocktail that converts cells neighboring damaged neurons into functional new neurons could potentially be used to treat stroke, Alzheimer's disease, and brain injuries.
Article word count: 702

HN Discussion: https://news.ycombinator.com/item?id=19191041
Posted by lettergram (karma: 6411)
Post stats: Points: 150 - Comments: 34 - 2019-02-18T15:00:20Z

\#HackerNews #cells #combo #creates #drug #from #neighboring #neurons #new

---

Article content:



A simple treatment using four small molecules converts human astrocytes - a common type of cells in the nervous system - into new neurons, which develop complex structures after four months, as pictured. Credit: Gong Chen Lab, Penn State

A simple drug cocktail that converts cells neighboring damaged neurons into functional new neurons could potentially be used to treat stroke, Alzheimerʼs disease, and brain injuries. A team of researchers at Penn State identified a set of four, or even three, molecules that could convert glial cells--which normally provide support and insulation for neurons--into new neurons. A paper describing the approach appears online in the journal [1]Stem Cell Reports on February 7, 2019."The biggest problem for brain repair is that neurons donʼt regenerate after brain damage, because they donʼt divide," said Gong Chen, professor of biology and Verne M. Willaman Chair in Life Sciences at Penn State and leader of the research team. "In contrast, glial cells, which gather around damaged brain tissue, can proliferate after brain injury. I believe turning glial cells that are the neighbors of dead neurons into new neurons is the best way to restore lost neuronal functions."Chenʼs team previously published research describing a sequence of nine small molecules that could directly convert human glial cells into neurons, but the large number of molecules and the specific sequence required for reprogramming the glial cells complicated the transition to a clinical treatment. In the current study, the team tested various numbers and combinations of molecules to identify a streamlined approach to the reprogramming of astrocytes, a type of glial cells, into neurons."We identified the most efficient chemical formula among the hundreds of drug combinations that we tested," said Jiu-Chao Yin, a graduate student in biology at Pen State who identified the ideal combination of small molecules. "By using four molecules that modulate four critical signaling pathways in human astrocytes, we can efficiently turn human astrocytes--as many as 70 percent--into functional neurons."The resulting chemically converted neurons can survive more than seven months in a culture dish in the lab. They form robust neural networks and send chemical and
electrical signals to each other, as normal neurons do inside the brain.Using three of the small molecules instead of four also results in the conversion of astrocytes into neurons, but the conversion rate drops by about 20 percent. The team also tried using only one of the molecules, but this approach did not induce conversion.Chen and his team had previously developed a gene therapy technology to convert astrocytes into functional neurons, but due to the excessive cost of gene therapy--which can cost a patient half a million dollars or more--the team has been pursuing more economical approaches to convert glial cells into neurons. The delivery system for gene therapies is also more complex, requiring the injection of viral particles into the human body, whereas the small molecules in the new method can be chemically synthesized and packaged into a pill."The most significant advantage of the new approach is that a pill containing small molecules could be distributed widely in the world, even reaching rural areas without advanced hospital systems," said Chen. "My ultimate dream is to develop a simple drug delivery system, like a pill, that can help stroke and Alzheimerʼs patients around the world to regenerate new neurons and restore their lost learning and memory capabilities."The researchers acknowledge that many technical issues still need to be resolved before a drug using small molecules could be created, including the specifics of drug packaging and delivery. They also plan to investigate potential side effects of this approach in future studies in order to develop the safest drug pills. Nonetheless, the research team is confident that this combination of molecules has promising implications for future drug therapies to treat individuals with neurological disorders."Our years of effort in discovering this simplified drug formula take us one step closer to reaching our dream," said Chen.

This article has been republished from [2]materials provided by [3]Penn State. Note: material may have been edited for length and content. For further information, please contact the cited source.

Reference: Jiu-Chao Yin, et al. Chemical Conversion of Human Fetal Astrocytes into Neurons through Modulation of Multiple Signaling Pathways. Stem Cell Reports. (2019) DOI: [4]https://doi.org/10.1016/j.stemcr.2019.01.003

References

Visible links
1. https://www.cell.com/stem-cell-reports/fulltext/S2213-6711(19)30004-9
2. http://science.psu.edu/news-and-events/2019-news/Chen2-2019
3. http://science.psu.edu/
4. https://doi.org/10.1016/j.stemcr.2019.01.003

HackerNewsBot debug: Calculated post rank: 111 - Loop: 221 - Rank min: 100 - Author rank: 31

Making My Own USB Keyboard from Scratch

A few months ago, I completed a project to build an entire USB keyboard from scratch. This included electronic circuit design, PCB design, firmware coding, CAD design, assembly and usage. The final…
Article word count: 1265

HN Discussion: https://news.ycombinator.com/item?id=19181473
Posted by blakesmith (karma: 176)
Post stats: Points: 142 - Comments: 37 - 2019-02-16T22:57:16Z

\#HackerNews #from #keyboard #making #own #scratch #usb

---

Article content:



A few months ago, I completed a project to build an entire USB keyboard from scratch. This included electronic circuit design, PCB design, firmware coding, CAD design, assembly and usage. The final result is my daily driver work keyboard, which I affectionately call “KeeBee”:

[1]KeeBee Keyboard, completed

A few project goals:

1. Implement the circuit myself
2. Write the keyboard firmware
3. Learn about how the USB protocol works

For my day job, I spend most of my time building cloud software that’s many layers removed from real running hardware. It’s extremely safisfying to peel back some of the abstractions, and get closer “to the metal” building real electronic devices I can physically touch and use.

Research & CAD Design

I knew I really liked the [2]OLKB Planck and Preonic style boards. They feature a nice minimal ortho-linear layout, that’s very compact. I also knew that I wanted to use Cherry MX Brown switches. With those two design components in mind, I started playing around with key layouts in [3]OpenSCAD. OpenSCAD is a great open source CAD design tool that functions more like a programming language than a WYSIWYG point and click mouse CAD tool.

Using the dimensions from a Cherry MX datasheet, I [4]hacked up a keyboard plate design, and then added switches and keycaps to get a feel for what the layout would look like. The top plate sits above the keyboard PCB and serves as a good switch stabilizer.

Top plate design:

[5]KeeBee CAD Plate

With keycaps added:

[6]KeeBee CAD Plate with Switches

[7]KeeBee CAD Plate with Switches

Prototype Circuit and Firmware Design

I chose an [8]STM32F042K6T6 as the main keyboard microcontroller. It’s around $3 per chip in individual quantities, and has just enough pins to implement a 69 [9]key scan matrix (32 pins in total). It sports an ARM Cortex M0 processor, and has a dedicated USB peripheral for sending out USB bits without tying up the main processor bit-banging out USB signals. I bought a [10]Nucleo prototype development board of this chip for experimenting with the chip before I integrated it into my PCB design. The Nucleo was easy to use on a breadboard, and power directly with USB.

I breadboarded out a small 4 key circuit, to test out the [11]diode based circuit I had researched. Ignoring the USB side of the equation, The first step was to just get the Cherry switches to reliably turn the 4 corresponding LEDs on and off when the button is pressed.

[12]KeeBee prototype circuit

Keyscan Matrix circuits are a technique to use when you have more switches than you have pins on your microcontroller.

After I got the [13]keyscan matrix implementation to my liking, it was time to work on the USB side of the equation.

The inner loop for the firmware is basically:

1. Scan all keys in the button keyscan matrix.
2. Map the button locations to their respective key symbols, using the currently selected layout (QWERTY, Dvorak, etc.)
3. Take the mappings, and generate USB HID Report packets and send them out the USB peripheral.
4. Set an LED on the keyboard to on if a key is pressed, off if not.

From [14]main.cc:

static void scan_and_update() { scan_matrix.Scan(key_scans, row_count, column_count); keyboard.SendReport( key_pipeline.MapKeyScans(key_scans, key_count)); update_key_press_status();
} int main() { Init(); status_led.SetOk(true); while (true) { scan_and_update(); }
}

The keyboard.SendReport component is the piece that actually sends the USB packets to the host. I struggled a lot to get USB working correctly. There are a lot of finnicky layers to the USB protocol that require accurate timing, and correct [15]device identification. I ended up needing to fire up Wireshark to sniff USB packets coming back and forth to my Linux laptop, in order to debug where things were getting lost on the wire. Most of my googling was pretty useless at this part of the build: suggestions I found had suggestions like, “You probably have a faulty USB device, you should get a new one.” When you’re the one actually trying to build the USB device, this isn’t really helpful. I was left reading through very large USB specifications that contain a lot of terminology that was pretty unfamiliar to me.

After mucking about for awhile, I was able to get the 4 key keyboard to correctly identify itself as a USB HID (Human Interface Device) to my laptop, and make sure all my key presses were being correctly mapped to the machine:

[16]KeeBee dmesg USB output

Getting a USB vendor and device id requires [17]paying a good bit of cash, so if you’re just doing something as a hobby, you’ll need to [18]hijack a similar device ID. I thought “Gear Head” sounded cool, and they make a keyboard, so I went with that one.

[19]KeeBee lsusb USB output

Schematic and PCB Design

With some working firmware on a working prototype, it’s time to put the schematic and PCB design into [20]KiCAD and get an actual printed circuit board made. Now that I had proven the schematic design worked, it was relatively straight-forward to connect everything together schematic wise:

[21]KeeBee Kicad Schematic

After building out the schematic and selecting part footprints, we need to layout the actual physical PCB:

[22]KeeBee Kicad PCB

KiCAD has a neat feature that lets you preview your PCB in 3D:

[23]KeeBee Kicad PCB 3D Front

[24]KeeBee Kicad PCB 3D Back

There are lots of great tutorials on how to use KiCAD. I started with Chris Gammell’s excellent [25]Getting to Blinkey 4.0 youtube series, where he takes you through building a LED blinker PCB circuit in KiCAD from start to finish.

PCB and Component Ordering

Once I was reasonably satisfied with the [26]schematic and PCB design, I started placing a bunch of orders:

1. All the board components from the keyboard’s [27]Bill of Materials: Switches, LEDs, diodes, microcontrollers, etc. I like to use [28]DigiKey for most of my electronic components.
2. The PCB itself. There are a lot of really great PCB prototype manufacturing services out there that will do small run PCB fabrication for really cheap. I’ve had great experience with [29]OshPark and [30]JLCPCB. For this project, I went with JLCPCB because of the board size cost, and because they let me pick a blue solder-mask board.
3. Any other cases, etc. For this project, my brother in law was able to laser cut the top and bottom keyboard plates from 1/4” acryllic sheets. There are other great online laser cutting and 3D printing services for case components, if you don’t have access to the equipment.

PCB arrival day is the the best:

[31]KeeBee PCB front

[32]KeeBee PCB back

JLCPCB is very affordable. This design was less than $30 shipped DHL from China, and took a little over a week to arrive after submitting my gerber files for order.

My brother in law took my [33]DXF files from OpenSCAD and tossed them in the laser cutter:

[34]KeeBee Laser Cutter

Final Assembly

With all the pieces ordered and fabricated, it was time to put the final keyboard together. I started with PCB component assembly: I used a soldering iron for the larger electronic components and a [35]hot air rework station for the small surface mount components like the STM32 microcontroller.

Total component build time for a board was roughly 3 hours - most of the time was spent soldering the 70 diodes and switches.

I added a JTAG debugger pin header to the PCB, which I used to plug in a [36]JLINK Edu mini to flash the microcontroller with the firmware with [37]OpenOCD.

From there, it was final testing, and plate assembly:

[38]KeeBee Final 1

[39]KeeBee Final 2

[40]KeeBee Final 3

My son thought it made a great train for his animals:

[41]KeeBee Train

[42]KeeBee Train Animals

Conclusion

From initial idea to final assembly, this project took about 3 months time. It was extremely rewarding to make something that I still use everyday at work.

All the project files are [43]up on GitHub, including firmware source code, PCB schematics, Bill of Materials, and CAD models.

Thanks for reading, and happy hardware hacking!

References

Visible links
1. http://blakesmith.me/images/keebee/keebee_1.jpg
2. https://olkb.com/
3. http://www.openscad.org/
4. https://github.com/blakesmith/embedded/blob/master/keebee/hardware/keyboard.scad
5. http://blakesmith.me/images/keebee/keebee_2_cad.png
6. http://blakesmith.me/images/keebee/keebee_3_cad.png
7. http://blakesmith.me/images/keebee/keebee_4_cad.png
8. https://www.digikey.com/product-detail/en/stmicroelectronics/STM32F042K6T6/497-14647-ND/4815294
9. https://en.wikipedia.org/wiki/Keyboard_matrix_circuit
10. https://www.digikey.com/product-detail/en/NUCLEO-F042K6/497-15980-ND/5428804
11. http://blog.komar.be/how-to-make-a-keyboard-the-matrix/
12. http://blakesmith.me/images/keebee/keebee_5_prototype.jpg
13. https://github.com/blakesmith/embedded/blob/master/drivers/stm32/scan_matrix.cc
14. https://github.com/blakesmith/embedded/blob/master/keebee/src/main.cc#L85
15. https://github.com/blakesmith/embedded/blob/master/keebee/src/usb_keyboard.cc#L12
16. http://blakesmith.me/images/keebee/keebee_6_dmesg.png
17. https://www.usb.org/getting-vendor-id
18. http://www.linux-usb.org/usb.ids
19. http://blakesmith.me/images/keebee/keebee_7_lsusb.png
20. http://kicad-pcb.org/
21. http://blakesmith.me/images/keebee/keebee_8_schematic.png
22. http://blakesmith.me/images/keebee/keebee_9_kicad_pcb.png
23. http://blakesmith.me/images/keebee/keebee_10_pcb_3d_front.png
24. http://blakesmith.me/images/keebee/keebee_11_pcb_3d_back.png
25.
26. https://github.com/blakesmith/embedded/tree/master/keebee/hardware
27. https://github.com/blakesmith/embedded/blob/master/keebee/bom.org
28. https://www.digikey.com/
29. https://oshpark.com/
30. https://jlcpcb.com/
31. http://blakesmith.me/images/keebee/keebee_12_pcb_front.jpg
32. http://blakesmith.me/images/keebee/keebee_13_pcb_back.jpg
33. https://github.com/blakesmith/embedded/blob/master/keebee/hardware/top_plate.dxf
34. http://blakesmith.me/images/keebee/keebee_14_laser_cutter.jpg
35. https://learn.sparkfun.com/tutorials/how-to-use-a-hot-air-rework-station/all
36. https://www.adafruit.com/product/3571
37. http://openocd.org/
38. http://blakesmith.me/images/keebee/keebee_15_final1.jpg
39. http://blakesmith.me/images/keebee/keebee_16_final2.jpg
40. http://blakesmith.me/images/keebee/keebee_17_final3.jpg
41. http://blakesmith.me/images/keebee/keebee_18_train.jpg
42. http://blakesmith.me/images/keebee/keebee_19_animals.jpg
43. https://github.com/blakesmith/embedded/tree/master/keebee

HackerNewsBot debug: Calculated post rank: 107 - Loop: 71 - Rank min: 100 - Author rank: 40

Most Americans don’t realize what companies can predict from their data

Every device that you use, every company you do business with, every online account you create – they all collect data about you and analyze it to figure out minute details of your life.
Article word count: 1300

HN Discussion: https://news.ycombinator.com/item?id=19178282
Posted by pseudolus (karma: 11581)
Post stats: Points: 97 - Comments: 73 - 2019-02-16T12:22:47Z

\#HackerNews #americans #can #companies #data #dont #from #most #predict #realize #their #what

---

Article content:




[1]Sixty-seven percent of smartphone users rely on Google Maps to help them get to where they are going quickly and efficiently.

A major of feature of Google Maps is its ability to predict how long different navigation routes will take. That’s possible because the mobile phone of each person using Google Maps sends data about its location and speed back to Google’s servers, where it is analyzed to generate new data about traffic conditions.

Information like this is useful for navigation. But the exact same data that is used to predict traffic patterns can also be used to predict other kinds of information – information people might not be comfortable with revealing.

For example, data about a mobile phone’s past location and movement patterns [2]can be used to predict where a person lives, who their employer is, where they attend religious services and the age range of their children based on where they drop them off for school.

These predictions label who you are as a person and guess what you’re likely to do in the future. Research shows that people are largely unaware that these predictions are possible, and, if they do become aware of it, [3]don’t like it. In my view, as someone who studies how predictive algorithms affect people’s privacy, that is a major problem for digital privacy in the U.S.

How is this all possible?

Every device that you use, every company you do business with, every online account you create or loyalty program you join, and even the government itself collects data about you.

The [4]kinds of data they collect include things like your name, address, age, Social Security or driver’s license number, purchase transaction history, web browsing activity, voter registration information, whether you have children living with you or speak a foreign language, the photos you have posted to social media, the listing price of your home, whether you’ve recently had a life event like getting married, your credit score, what kind of car you drive, how much you spend on groceries, how much credit card debt you have and the location history from your mobile phone.

It doesn’t matter if these datasets were collected separately by different sources and don’t contain your name. It’s still easy to match them up according to other information about you that they contain.

For example, there are identifiers in public records databases, like your name and home address, that can be matched up with GPS location data from an app on your mobile phone. This allows a third party to link your home address with the location where you spend most of your evening and nighttime hours – presumably where you live. This means the app developer and its partners have access to your name, even if you didn’t directly give it to them.

In the U.S., [5]the companies and platforms you interact with own the data they collect about you. This means they can legally sell this information to data brokers.

Data brokers are companies that are in the business of buying and selling datasets from a wide range of sources, including location data from [6]many mobile phone carriers. Data brokers combine data to create detailed profiles of individual people, which they [7]sell to other companies.

Combined datasets like this can be used to predict what you’ll want to buy in order to target ads. For example, a company that has purchased data about you can do things like connect your social media accounts and web browsing history with the route you take when you’re running errands and your purchase history at your local grocery store.

Employers use large datasets and predictive algorithms to make decisions about who to interview for jobs and [8]predict who might quit. Police departments make lists of people who may be [9]more likely to commit violent crimes. FICO, the same company that calculates credit scores, also calculates a [10]“medication adherence score” that predicts [11]who will stop taking their prescription medications.

[12] [IMG]Research shows that people are only aware of predictions that are shown to them in an app’s user interface, and that make sense given the reason they decided to use the app. [13]SIFO CRACHO/shutterstock.com

How aware are people about this?

Even though people may be aware that their mobile phones have GPS and that their name and address are in a public records database somewhere, it’s far less likely that they realize [14]how their data can be combined to make new predictions. That’s because privacy policies typically only include [15]vague language about how data that’s collected will be used.

[16]In a January survey, the Pew Internet and American Life project asked adult Facebook users in the U.S. about the predictions that Facebook makes about their personal traits, based on data collected by the platform and its partners. For example, Facebook assigns a “multicultural affinity” category to some users, guessing how similar they are to people from different race or ethnic backgrounds. This information is used to target ads.

The survey found that 74 percent of people did not know about these predictions. About half said they are not comfortable with Facebook predicting information like this.

[17]In my research, I’ve found that people are only aware of predictions that are shown to them in an app’s user interface, and that makes sense given the reason they decided to use the app. For example, a [18]2017 study of fitness tracker users showed that people are aware that their tracker device collects their GPS location when they are exercising. But this doesn’t translate into awareness that the activity tracker company can predict where they live.

In another study, I found that Google Search users know that Google collects data about their search history, and Facebook users are aware that Facebook knows who their friends are. But [19]people don’t know that their Facebook “likes” can be used to [20]accurately predict their political party affiliation or sexual orientation.

What can be done about this?

Today’s internet largely relies on people managing their own digital privacy.

Companies ask people up front to consent to systems that collect data and make predictions about them. [21]This approach would work well for managing privacy, if people refused to use services that have privacy policies they don’t like, and if companies wouldn’t violate their own privacy policies.

But research shows that [22]nobody reads or understands those privacy policies. And, even when companies face consequences for breaking their privacy promises, it doesn’t stop them from [23]doing it again.

Requiring users to consent without understanding how their data will be used also allows companies to shift the blame onto the user. If a user starts to feel like their data is being used in a way that they’re not actually comfortable with, they don’t have room to complain, because they consented, right?

In my view, there is no realistic way for users to be aware of the kinds of predictions that are possible. People naturally expect companies to use their data only in ways that are related to the reasons they had for interacting with the company or app in the first place. But companies usually aren’t legally required to restrict the ways they use people’s data to only things that users would expect.

One exception is Germany, where the Federal Cartel Office [24]ruled on Feb. 7 that Facebook must specifically ask its users for permission to combine data collected about them on Facebook with data collected from third parties. The ruling also states that if people do not give their permission for this, they should still be able to use Facebook.

I believe that the U.S. needs stronger privacy-related regulation, so that companies will be more transparent and accountable to users about not just the data they collect, but also the kinds of predictions they’re generating by combining data from multiple sources.

References

Visible links
1. https://themanifest.com/app-development/popularity-google-maps-trends-navigation-apps-2018
2. https://www.nytimes.com/interactive/2018/12/10/business/location-data-privacy-apps.html
3. https://repository.upenn.edu/asc_papers/398/
4. https://www.ftc.gov/system/files/documents/reports/data-brokers-call-transparency-accountability-report-federal-trade-commission-may-2014/140527databrokerreport.pdf
5. https://harvardlawreview.org/2017/05/if-these-walls-could-talk-the-smart-home-and-the-fourth-amendment-limits-of-the-third-party-doctrine/
6. https://www.wired.com/story/carriers-sell-location-data-third-parties-privacy/
7. https://motherboard.vice.com/en_us/article/bjpx3w/what-are-data-brokers-and-how-to-stop-my-private-data-collection
8. https://www.businessinsider.com/workday-predicts-when-employees-will-quit-2014-11
9. https://bigdata.fairness.io/predictive-policing/
10. https://www.fico.com/en/products/fico-medication-adherence-score
11. https://www.fico.com/en/resource-download-file/3630
12. https://images.theconversation.com/files/257557/original/file-20190206-174861-1w2r23i.jpg?ixlib=rb-1.1.0&q=45&auto=format&w=1000&fit=clip
13. https://www.shutterstock.com/image-photo/closeup-shot-bearded-sportive-man-after-485402515?src=jfwCSwgeTOnU54CqULK6uA-1-16
14. http://dx.doi.org/10.14722/usec.2016.23017
15. https://doi.org/10.1086/688669
16. http://www.pewinternet.org/2019/01/16/facebook-algorithms-and-personal-data/
17. https://scholar.google.com/citations?user=uXpvv2V2xKkC&hl=en
18. https://www.usenix.org/conference/soups2017/technical-sessions/presentation/rader
19. https://www.usenix.org/conference/soups2014/proceedings/presentation/rader
20. https://doi.org/10.1073/pnas.1218772110
21. https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2171018
22. https://doi.org/10.1080/08838151.2018.1451867
23. https://www.recode.net/2019/1/23/18193314/facebook-ftc-investigation-explained-privacy-agreement
24. https://www.bundeskartellamt.de/SharedDocs/Publikation/EN/Pressemitteilungen/2019/07_02_2019_Facebook.pdf?__blob=publicationFile&v=2

HackerNewsBot debug: Calculated post rank: 89 - Loop: 133 - Rank min: 80 - Author rank: 74

Moving from Ruby to Rust

How we migrated our Tier 1 service from ruby to rust and didn’t break production.
Article word count: 42

HN Discussion: https://news.ycombinator.com/item?id=19166020
Posted by steveklabnik (karma: 59589)
Post stats: Points: 122 - Comments: 89 - 2019-02-14T21:06:14Z

\#HackerNews #from #moving #ruby #rust

---

Article content:




Posted by [1]Andrii Dmytrenko on Thursday, February 14, 2019

In the Logistics Algorithms team, we have a service, called Dispatcher, the main purpose of which is to offer an order to the rider, optimally. For each rider we build a timeline, where we predict where riders will be at a certain point of time; knowing this, we can more efficiently suggest a rider for an order.

Building each timeline involves a fair bit of computation: using different machine learning models to predict how long events will take, asserting certain constraints, calculating assignment cost. The computations themselves are quick, but the problem is that we need to do a lot of them: for each order, we need to go over all available riders to determine which assignment would be the best.

The first version of the Dispatcher was written mainly in Ruby: this was a go-to language in the company, and it was performing adequately given our size at the time. However, as Deliveroo kept growing, the number of orders and riders increaed dramatically, and we saw that the dispatch process started taking much longer than before and we realised, that at some point it will be impossible to dispatch some areas within a time constraint that we put in place. We also knew that is was limiting us if we decided to implement more advanced algorithms, which would require even more computation time.

The first thing we tried was to optimise the current code (cache some computations, try to find a bug in the algorithms), which didn’t help much. It was clear that Ruby was a bottleneck here and we started looking at the alternatives.

Why Rust?

We considered a few approaches to how to solve the problem of dispatch speed:
\* choose a new programming language with better performance characteristics and rewrite the Dispatcher
\* identify biggest bottlenecks, rewrite those parts of the code and somehow integrate them in the current code
We knew that rewriting something from scratch is risky, as it can introduce bugs, and switching services over can be painful, so we didn’t feel quite comfortable with this approach. Another option, finding bottlenecks and replacing them, was something that we did already for one part of the code (we built a native extension gem for the Hungarian route matching algorithm, implemented in Rust), and that worked well. We decided to try this approach.

There were several options how we could integrate parts of the code written in another language to work with Ruby:
\* build an external service and provide an API to communicate with
\* build a native extension
We quickly discarded an option to build an external service, because either we would need to call this external service hundreds of thousands of times per dispatch cycle and the overhead of the communication would offset all of the potential speed gains, or we would need to reimplement a big part of the dispatcher inside this service, which is almost the same as a complete rewrite.

We decided that it has to be some sort of native extension, and for that, we decided to use Rust, as it ticked most of the boxes for us:
\* it has high performance (comparable to C)
\* it is memory safe
\* it can be used to build dynamic libraries, which can be loaded into Ruby (using extern "C" interface)
Some of our team members had experience with Rust and liked the language, also one part of the Dispatcher was already using Rust. Our strategy was to replace the current ruby implementation gradually, by replacing parts of the algorithm one by one. It was possible because we could implement separate methods and classes in Rust and call them from Ruby without a big overhead of cross-language interaction.

How we made Ruby talk to Rust

There a few different ways you can call Rust from Ruby:
\* write a dynamic library in Rust with extern "C" interface and call it using [2]FFI.
\* write a dynamic library, but use the Ruby API to register methods, so that you can call them from Ruby directly, just like any other Ruby code.
The first approach, using FFI would require us to come up with some custom C like interfaces in both Rust and Ruby and then create wrappers for them in both languages. The second approach, using Ruby API, sounded more promising, as there were already libraries to make our lives easier:

We tried Helix first:
\* it has macros which look like writing Ruby in Rust, which was a bit more magical for us than we were comfortable with
\* the Coercion Protocol wasn’t well documented and it wasn’t clear how would you go about passing non-primitive Ruby objects into Helix methods
\* we were not sure about the safety - it looked like Helix didn’t call Ruby methods using [3]rb_protect, which could lead to undefined behavior
Eventually, we decided to go with ruru/rutie, but keep the Ruby layer thin and isolated so that we could possibly switch in the future. We decided to use [4]Rutie, a recent fork of [5]Ruru which has more active development.

Here’s a small example of how you can create a class with one method in ruru/rutie:

#
[macro_use]extern crate rutie; use rutie::{Class, Object, RString}; class!(HelloWorld);
methods!( HelloWorld, _itself, fn hello(name: RString) -> RString { RString::new(format!("Hello {}", name.unwrap().to_string())) }
); #[allow(non_snake_case)]
#
[no_mangle]pub extern "C" fn Init_ruby_rust_demo() { let mut class = Class::new("RubyRustDemo", None); class.define(|itself| itself.def_self("hello", hello) );
}

It’s great if all you need is to pass some basic types (like String, Fixnum, Boolean, etc.) to your methods, but not that great if you need to pass a lot of data. In that case, you can pass the whole object, say Order and then you would need to call each field you need on that object to move it into Rust:

pub struct RustUser { name: String, address: Address,
} pub struct Address { pub country: String, pub city: String,
} class!(User); impl VerifiedObject for User { fn is_correct_type

You can see a lot of routine and repetitive code here, proper error handling is missing as well. After looking at this code, it reminded us that this looks a lot like some manual parsing of something like JSON or similar. You could instead serialize objects in Ruby to JSON and then parse it in Rust, and it works mostly OK, but you still need to implement JSON serializers in Ruby. Then we were curious, what if we implement serde deserializer for AnyObject itself: it will take ruties’s AnyObject and go over each field defined in the type and call the corresponding method on that ruby object to get it’s value. It worked!

Here’s the same method, but using our serde deserializer & serializer:

#[derive(Debug, Deserialize)]
pub struct User { pub name: String, pub address: Address,
} #[derive(Debug, Deserialize)]
pub struct Address { pub country: String, pub city: String
} class!(HelloWorld);
rutie_serde_methods!( HelloWorld, _itself, ruby_class!(Exception), // Notice that the argument has our defined type User, and the return type is plain bool fn hello_user(user: User) -> bool { do_something_with_user(&user); true }
);

You can see how much simpler the code in hello_user is now - we don’t need to parse user manually anymore. Since it’s serde, it can also handle nested objects (as you can see with the address). We also added a built-in error handling: if serde fails to “parse” the object, this macro will raise an exception of a class that we provided (Exception in this case), it also wraps the method body in the [6]panic:ch_unwind, and re-raises panics as exceptions in Ruby.

Using [7]rutie-serde we could quickly and painlessly implement thin interfaces between ruby and rust.

We came up with a plan to gradually replace all parts of the Ruby Dispatcher with Rust. We started by replacing with Rust classes which didn’t have dependencies on other parts of the Dispatcher and adding feature flags, something similar to this:

module TravelTime def self.get(from_location, to_location, options) # in the real world the feature flag would be more granular and enable you to do an incremental roll-out if rust_enabled? && Feature.enabled?(:rust_travel_time) RustTravelTime.get(from_location, to_location, options) else RubyTravelTime.get(from_location, to_location, options) end end
end

There was also a master switch (in this case rust_enabled?), which allowed us to switch all the Rust code off by flipping just one feature flag.

Since the API of both Ruby and Rust classes implementations remained largely the same, we were able to test both of them using the same tests, which gave us more confidence in the quality of the implementation.

RSpec.describe TravelTime do shared_examples "travel_time" do let(:from_location) { build(:location) } let(:to_location) { build(:location) } let(:options) { build(:travel_time_options) } it ʼreturns correct travel timeʼ do expect(TravelTime.get(from_location, to_location, options)).to eq(123.45) end end context "ruby implementation" do before do Feature.disable!(:rust_travel_time) end include "travel_time" end context "rust implementation" do before do Feature.enable!(:rust_travel_time) end include "travel_time" end
end

It was also very important that, at any time, we could switch off the Rust integration and the Dispatcher would still work (because we kept the Ruby implementation along with Rust and kept adding feature flags).

Performance Improvements

When moving more larger chunks of code into Rust, we noticed increased performance improvements which we were carefully monitoring. When moving smaller modules to Rust, we didn’t expect much improvement: in fact, some code became slower because it was being called in tight loops, and there was a small overhead to calling Rust code from the Ruby application.

Performance numbers

In the Dispatcher, there are 3 main phases of the dispatch cycle:
\* loading data
\* running computation, calculating assignments
\* saving/sending assignments
Loading data and saving data phases scale pretty much linearly depending on the dataset size, while the computation phase (which we moved to Rust) has an higher-order polynomial component in it. We are less worried about the loading/saving data phases, and we didn’t prioritise speeding up those phases yet. While loading data and sending data back were still parts of the Dispatcher written in Ruby, the total dispatch time was significantly reduced: for example, in one of our larger zones it dropped from ~4 sec to 0.8 sec.

[8]Total dispatch time

Out of those 0.8 seconds, roughly 0.2 seconds were spent in Rust, in the computation phase. This means 0.6 second is a Ruby/DB overhead of loading data and sending assignments to riders. It looks like the dispatch cycle is only 5 times quicker now, but actually, the computation phase in this example time was reduced from ~3.4sec to 0.2sec, which is a 17x speedup.

[9]Computation phase time

Keep in mind, that Rust code is almost a 1:1 copy of the Ruby code in terms of the implementation, and we didn’t add any additional optimisations (like caching, avoiding copying memory in some cases), so there is still room for improvement.

Conclusion

Our project was successful: moving from Ruby to Rust was a success that dramatically sped up our dipatch process, and gave us more head-room in which we could try implementing more advanced algorithms.

The gradual migration and careful feature flagging mitigated most of the risks of the project. We were able to deliver it in smaller, incremental parts, just like any other feature that we normally build in Deliveroo.

Rust has shown a great performance and the absence of runtime made it easy to use it as a replacement of C in building Ruby native extensions.

References

Visible links
1. http://deliveroo.engineering/authors/andrii-dmytrenko
2. https://github.com/ffi/ffi/wiki
3. https://silverhammermba.github.io/emberb/c/#rb_protect
4. https://crates.io/crates/rutie
5. https://crates.io/crates/ruru
6. https://doc.rust-lang.org/beta/std/panic/fn.catch_unwind.html
7. https://crates.io/crates/rutie-serde/

HackerNewsBot debug: Calculated post rank: 111 - Loop: 164 - Rank min: 100 - Author rank: 49

US iPhone users spent, on average, $79 on apps last year, up 36% from 2017

Apple’s push to get developers to build subscription-based apps is now having a notable impact on App Store revenues. According to a new report from Sensor Tower due out later this week, revenue…
Article word count: 622

HN Discussion: https://news.ycombinator.com/item?id=19139779
Posted by kjhughes (karma: 9206)
Post stats: Points: 121 - Comments: 61 - 2019-02-12T00:04:42Z

\#HackerNews #2017 #apps #average #from #iphone #last #spent #users #year

---

Article content:




Apple’s push to get developers to build subscription-based apps is now having a notable impact on App Store revenues. According to a new report from [1]Sensor Tower due out later this week, revenue generated per U.S. iPhone grew 36 percent, from $58 in 2017 to $79 last year. As is typical, much of that increase can be attributed to mobile gaming, which accounted for more than half of this per-device average. However, more substantial growth took place in the categories outside of gaming — including those categories where subscription-based apps tend to rule the top charts, the firm found.

According to the report’s findings, per-device app spending in the U.S. grew more over the past year than it did in 2017.

From 2017 to 2018, iPhone users spent an average of $21 or more on in-app purchases and paid app downloads — a 36 percent increase compared with the 23 percent increase from 2016 to 2017, when revenue per device grew from $47 to $58.

However, 2018’s figure was slightly lower than the 42 percent increase in average per-device spending seen between 2015 and 2016, when revenue grew from $33 to $47, noted Sensor Tower.

As usual, mobile gaming continued to play a large role in iPhone spending. In 2018, gaming accounted for nearly 56 percent of the average consumer spend — or $44 out of the total $79 spent per iPhone.

But what’s more interesting is how the non-gaming categories fared this past year.

Some categories — including those where subscription-based apps dominate the top charts — saw even higher year-over-year growth in 2018, the firm found.

For example, Entertainment apps grew their spend per device increase by 82 percent to $8 of the total in 2018. Lifestyle apps increased by 86 percent to reach $3.90, up from $2.10.

And though it didn’t make the top five, Health & Fitness apps also grew 75 percent year-over-year to account for an average of $2.70, up from $1.60 in 2017.

Other categories in the top five included Music and Social Networking apps, which both grew by 22 percent.

This data indicates that subscription apps are playing a significant role in helping drive iPhone consumer spending higher.

The news comes at a time when Apple has [2]reported [3]slowing iPhone sales, which is pushing the company to [4]lean more on services to continue to boost its revenue. This includes not just App Store subscriptions, but also things like Apple Music, Apple Pay, iCloud, App Store Search ads, AppleCare and more.

As subscriptions become more popular, Apple will need to remain vigilant against those who would abuse the system.

For example, a number of sneaky subscription apps [5]were found plaguing the App Store in recent weeks. They were duping users into paid memberships with tricky buttons, hidden text, instant trials that converted in days and the use of other misleading tactics.

Apple later cracked down by removing some of the apps, and [6]updated its developer guidelines with stricter rules about how subscriptions should both look and operate.

A failure to properly police the App Store or set boundaries to prevent the overuse of subscriptions could end up turning users off from downloading new apps altogether — especially if users begin to think that every app is after a long-term financial commitment.

Developers will need to be clever to convert users and retain subscribers amid this shift away from paid apps to those that come with a monthly bill. App makers will need to [7]properly market their subscription’s benefits, and even consider offering bundles to increase the value.

But in the near-term, the big takeaway for developers is that there is still good money to be made on the App Store, even if iPhone sales are slowing.

References

Visible links
1. https://sensortower.com/
2. https://techcrunch.com/2019/01/29/yep-iphone-revenue-is-down/
3. https://www.businessinsider.com/apple-will-no-longer-report-iphone-sales-unit-numbers-2018-11
4. https://www.bloomberg.com/news/articles/2019-01-30/services-growth-is-next-in-sight-for-apple-as-iphone-sales-drop
5. https://techcrunch.com/2018/10/15/sneaky-subscriptions-are-plaguing-the-app-store/
6. https://techcrunch.com/2019/01/28/apples-new-developer-guidelines-signal-that-scammy-subscription-apps-time-is-up/
7. https://techcrunch.com/2018/05/06/subscription-hell/

HackerNewsBot debug: Calculated post rank: 101 - Loop: 167 - Rank min: 100 - Author rank: 68

Russia considers 'unplugging' from internet

Russia may briefly disconnect from the internet as part of a test of its cyber-defences.
Article word count: 623

HN Discussion: https://news.ycombinator.com/item?id=19135085
Posted by _void (karma: 73)
Post stats: Points: 114 - Comments: 85 - 2019-02-11T15:29:23Z

\#HackerNews #considers #from #internet #russia #unplugging

---

Article content:




[1]Vladimir Putin Image copyright Reuters Image caption The net independence plan is seen as a way for Russiaʼs government to get more control over online life

Russia is considering whether to disconnect from the global internet briefly, as part of a test of its cyber-defences.

The test will mean data passing between Russian citizens and organisations stays inside the nation rather than being routed internationally.

A draft law mandating technical changes needed to operate independently was introduced to its parliament last year.

The test is expected to happen before 1 April but no exact date has been set.

Major disruption

The draft law, called the Digital Economy National Program, requires Russiaʼs ISPs to ensure that it can operate in the event of foreign powers acting to isolate the country online.

Nato and its allies have threatened to sanction Russia over the cyber-attacks and other online interference which it is regularly accused of instigating.

The measures outlined in the law include Russia building its own version of the netʼs address system, known as DNS, so it can operate if links to these internationally-located servers are cut.

Currently, 12 organisations oversee the root servers for DNS and none of them are in Russia. However many copies of the netʼs core address book do already exist inside Russia suggesting its net systems could keep working even if punitive action was taken to cut it off.

The test is also expected to involve ISPs demonstrating that they can direct data to government-controlled routing points. These will filter traffic so that data sent between Russians reaches its destination, but any destined for foreign computers is discarded.

Eventually the Russian government wants all domestic traffic to pass through these routing points. This is believed to be part of an effort to set up a mass censorship system akin to that seen in China, which tries to scrub out prohibited traffic.

Russian news organisations reported that the nationʼs ISPs are broadly backing the aims of the draft law but are divided on how to do it. They believe the test will cause "major disruption" to Russian internet traffic, [2]reports tech news website ZDNet.

The Russian government is providing cash for ISPs to modify their infrastructure so the redirection effort can be properly tested.

Analysis: Zoe Kleinman, BBC technology reporter

How does an entire country "unplug" itself from the internet?

Itʼs important to understand a little about how the internet works. It is essentially a series of thousands of digital networks along which information travels. These networks are connected by router points - and they are notoriously the weakest link in the chain.

What Russia wants to do is to bring those router points that handle data entering or exiting the country within its borders and under its control- so that it can then pull up the drawbridge, as it were, to external traffic if itʼs under threat - or if it decides to censor what outside information people can access.

Chinaʼs firewall is probably the worldʼs best known censorship tool and it has become a sophisticated operation. It also polices its router points, using filters and blocks on keywords and certain websites and redirecting web traffic so that computers cannot connect to sites the state does not wish Chinese citizens to see.

It is possible to get around some firewalls using virtual private networks (VPNs) - which disguise the location of a computer so the filters do not kick in - but some regimes are more tolerant of them than others. [3]China cracks down on them from time to time and the punishment for providing or using illegal VPNs can be a prison sentence.

Occasionally countries disconnect themselves by accident - Mauritania was left offline for two days in 2018 after the undersea fibre cable that supplied its internet was cut, possibly by a trawler.

References

Visible links
2. https://www.zdnet.com/article/russia-to-disconnect-from-the-internet-as-part-of-a-planned-test/
3. https://www.bbc.co.uk/news/technology-40960423

HackerNewsBot debug: Calculated post rank: 104 - Loop: 172 - Rank min: 100 - Author rank: 104

Making Money from Open-Source Software

After my previous two posts on the topic, it is now time to discuss how we make money from Open Source Software. Before I start, I want to clarify what I’m talking about:RavenDB is a document…
Article word count: 1231

HN Discussion: https://news.ycombinator.com/item?id=19123153
Posted by aliswe (karma: 59)
Post stats: Points: 111 - Comments: 32 - 2019-02-09T16:37:12Z

\#HackerNews #from #making #money #open-source #software

---

Article content:



[1]imageAfter my previous [2]two [3]posts on the topic, it is now time to discuss how we make money from Open Source Software. Before I start, I want to clarify what I’m talking about:
\* RavenDB is a document database.
\* It is about a decade old.
\* The server is released under the AGPL / commercial license.

\* We offer free community / developer licenses without any AGPL hindrance.

\* The RavenDB client APIs are licensed under the MIT license.
\* RavenDB (the product) is created by Hibernating Rhinos (the company).
I created RavenDB because I couldn’t not to. It was an idea that had to go out of my head. I looked up the details, and toward the [4]end of 2008 I started to work on it as a side project. At the time I was involved in five or six active open source projects, just got my NHibernate Profiler product to a stable ground and was turning the idea of getting deeper into databases in my head for a while. So I sat down and wrote some code.

I was just doing some code doodling and it turned into [5]deep design discussion and at some point I was actually [6]starting to actively look for hep building the user interface for a “done” project. That was in late Feb 2010. Somehow, throwing some code at the compiler become over a journey that lasted over a year in which I worked 16+ hours days on this project.

Around Mar 2010 I knew that I had a problem. Continuing as I did before, just writing a lot of code and trying to create an OSS project out of it would eat up all my time (and money). The alternatives were actually making money from RavenDB or stop working on it completely. And I didn’t want to stop working on it.

I decided that I had to make an effort to actually make a product out of this project. And that meant that I had to sit down and plan how I would actually make money from it. I firmly believe that “build it, and they will come” is a nice slogan, but it doesn’t replace planning, strategy and (at least some) luck.
\* I already knew that I couldn’t sustain the project as a labor of love, and donations are not a sustainable way (or indeed, a way) to make money.
\* Sponsorship seemed like it would be unlikely unless I got one of my clients to start using RavenDB and then have them pay me to maintain it. That seemed… unethical, so wasn’t an option.
\* Services / consulting was something that I was already doing quite heavily, and was quite successful at it. But this is a labor intensive way of making money and it would compete directly with the time that it would take to build RavenDB itself.
\* Support is a model I really don’t like, because it put me in a conflict of interest. I take pride in what I do, and I wanted to make something that would be easy to use and not require support.
\* Open Core / N versions back – are models that I don’t like. The open core model often leaves out critical functionality (such as security) and the N versions back mean that you give the users you most want to have the best experience (since that would encourage them to give you money) the worst experience (here are all our bugs that we fixed but won’t give to you yet).
That left us with dual licensing as a way to make money. I chose the AGPL because it was an OSI approved license that isn’t friendly for commercial use, leading most users who want to use it to purchase a commercial license.

So far, this is fairly standard, I believe.

I decided that RavenDB is going to be OSS, but from most other aspects, I’m going to treat it as a commercial product. It had a paid team working on it from the moment it stopped being a proof of concept. It meant that we are intentionally set out to make our money on the license. This, in turn had a lot of implications. Support is defined as a Cost Center in Hibernating Rhinos. In other words, one of the things that we routinely do in Hibernating Rhinos is look at how we can reduce support.

One way of doing that, of course, is not have support, or staff the support team with students or the cheapest off shore option available. Instead, our support staff consists of decided support engineers and the core team that builds RavenDB. This serves several goals. First, it means that when you raise a support issue with us, you get someone who knows what they are doing. Second, it means that the core team is directly exposed (and affected by) the support issues that are raised. I have structured things in this manner explicitly because having an insight into actual deployment and customer behavior means that the team is directly aware of the impact of their work. For example, writing an error message that will explain some issue to the user matters, because it would reduce the time an engineer spends on the phone troubleshooting (not fun) and increases the amount of time they can sling code around (fun).

We had a major update between versions 3.5 and 4.0, taking almost 3 years to finish. The end result was a vastly improved performance, the ability to run on multiple platforms and a whole host of other cool stuff. But the driving force behind it all? We had to make a significant change to our architecture in order for us to reduce the support burden. It worked, and the need for support went down by over 80%.

Treating RavenDB as a commercial product from the get go, even though it had an OSS license, meant that we focused on a lot of the stuff that is mostly boring. Anything from docs, setup and smoothing out all the bumps in the road, etc. The AGPL was there as a way to have your cake and eat it too. Be an OSS project with all the benefits that this entails. Confidence from our users about what we do, entry to the marketplace, getting patches from users and many more. Just having the ability to directly talk to our community with the code in front of all of us has been invaluable.

At the same time, we sell licenses to RavenDB, which is how we make money. The idea is that we provide value above and beyond whatever it is our license cost, and we can do that because we are very upfront and obvious in how we get paid.

We have a few users who have chosen to go with the AGPL version and skip paying us. I would obviously rather get paid, but I have laid out the rules of the game when I started playing and that is certainly within the rules. I believe that we’ll meet these users as customers in the future, it isn’t really that different from the community edition which we offer freely. In both cases, we aren’t getting paid, but it expands our reach, which will usually get us more customers in the long run.

We have been doing this for a decade and Hibernating Rhinos currently has about 30 people working full time on it, so it is certainly working so far [7]Smile!

References

Visible links
1. https://ayende.com/blog/Images/Open-Live-Writer/Making-money-from-Open-Source-Software-H_147F4/image_2.png
2. https://ayende.com/blog/186113-A/making-money-from-open-source-software-the-problem?key=f2a4ae9e5c04431f8aa3a4ef56822869
3. https://ayende.com/blog/186146-A/making-money-from-open-source-software-the-dichotomy?key=6822254250dc4546964dc7b15e29095f
4. https://ayende.com/blog/3768/longest-time-to-first-test-pass-but-it-now-works
5. https://ayende.com/blog/posts/series/3898/designing-a-document-database
6. https://ayende.com/blog/4413/user-interface-delegation-rhino-divan-db

HackerNewsBot debug: Calculated post rank: 84 - Loop: 234 - Rank min: 80 - Author rank: 11

From Google Analytics to Fathom

tl;dr: I'm now using Fathom for my personal website analytics, and it's easy to self-host and maintain, better for privacy, and can lead to better site performance. Since the mid-2000s, right after it…
Article word count: 2159

HN Discussion: https://news.ycombinator.com/item?id=19119868
Posted by geerlingguy (karma: 4617)
Post stats: Points: 124 - Comments: 39 - 2019-02-09T00:27:16Z

\#HackerNews #analytics #fathom #from #google

---

Article content:



tl;dr: Iʼm now using Fathom for my personal website analytics, and itʼs easy to self-host and maintain, better for privacy, and can lead to better site performance.
Since the mid-2000s, right after it became available, I started using [1]Google Analytics for almost every website I built (whether it be mine or someone else). It quickly became (and remains) the de-facto standard for website usage analytics and user tracking.

[2]Google Analytics UI

Before that you basically had web page visit counters (some of them with slightly more advanced features ala [3]W3Counter and [4]Stat Counter), and then on the high end you had Urchin Web Analytics (which is what Google acquired and turned into a ʼcloudʼ version, naming the new product Google Analytics and tying it deeply into the Google AdWords ecosystem).

[5]Urchin Web Analytics Logo
With such a fun logo, how could Google resist acquiring it?

The amount of data you can get out of GA is pretty amazing, and you can add custom event tracking, track website goals and conversion rates, and program your own metrics and events. But this all comes at a cost; sure, Google Analytics is free, but Google uses the gobs of data about Internet usage it gets from [6]tracking half the Internet for itʼs own purposes (much like it uses free-to-upload Google Photo data for AI training purposes). Not only that, trying to get just the data that I care about out of Google Analytics can be a time sink. Plus, I have to disable my ad blocker every time I go to view analytics!

So in the interest of going back to a simpler time, when I ran my own Urchin server and tracked analytics locally, Iʼve decided to give [7]Fathom a chance, and started sending all the analytics for my personal and SaaS websites to a new Fathom server:

[8]Fathom Analytics UI

In this blog post, Iʼll document how I did it, how itʼs working out, and compare it to Google Analytics. Maybe by the end youʼll want to try it out, too!

Building a reproducible Fathom server

Many people know me as ʼthe Ansible guyʼ—either for my [9]book, or for one of a zillion posts Iʼve written on Ansible (mostly for my own memory, but luckily theyʼre helpful for others too!). I noticed that there so far was no Ansible role on Galaxy for Fathom, so I went ahead and added [10]geerlingguy.fathom.

Using that role (alongside a few others), you can install a production-grade Fathom instance in just a few lines of YAML:

---- hosts: analytics become: true vars_files: - vars/main.yml roles: - geerlingguy.repo-epel - geerlingguy.security - geerlingguy.firewall - geerlingguy.fathom - geerlingguy.certbot - geerlingguy.nginx

- geerlingguy.git

I set the appropriate variables for fathom, nginx, and certbot in vars/main.yml, but other than that, itʼs a tiny Ansible playbook (this is literally the playbook I am using, right now, for that server), and I coupled that with another short provisioning playbook which builds a Digital Ocean 1 GB VPS running CentOS 7 and adds a user account for me, and in less than 1.5 hours I went from never having downloaded Fathom to having a nice little production-grade Fathom analytics server!

If you want to use my Ansible role to build your own Fathom server, Iʼd encourage you to check out the [11]roleʼs documentation and try it yourself. Fathom also offers an open source [12]Docker image you can use if you want to deploy it even more easily. In fact, if you have Docker installed and want to try Fathom in less than a minute, just run:

docker run -d -p 8080:8080 usefathom/fathom:latest

Then visit http://localhost:8080/ enter the domain for a website you want to track. Note that you would only be able to track websites which can hit your localhost, and the data stored would be wiped out when you remove the container (the above is just for testing).

Using Fathom

One thing I look for in software (especially if I donʼt have time to dive too deep into the source code) is simplicity—is the initial experience well-though-out, and easy to understand, or are there a lot of roadblocks?

Fathom ticks off the box for both being dead-simple to install and run (most Go apps seem to be, since theyʼre just binaries you download), and easy to get started tracking your first website (it comes up and asks you for a domain to track, then shows a box where you can copy out the tracking code to put on the site).

There are a few little UI issues Iʼve found which will likely get ironed out in coming months:
\* Itʼs not obvious that you canʼt add more sites until youʼve created a user account (command line only right now)
\* There are animations which occur on graphs and data every time you click on things (this will be able to be turned off soon)
\* The UI has a few elements which arenʼt very accessible (e.g. font size is small and it has grey-on-grey... it looks like they copied one bad feature from Google Analytics 🙄).
But overall the UI is consistent, and didnʼt take long at all to master (especially owing to its spartan feature set).

Features

In its current state, Fathom is almost the polar opposite to a tool like Google Analytics. Whereas in GA you can quickly get lost eight levels deep in a certain metric, Fathom can feel quite spartan in the fact that there isnʼt yet a concept of ʼdrilling downʼ.

Right now the UI exposes a few things:
\* Date selection
\* Hourly or daily graph frequency
\* Basic metrics (unique visits, pageviews, avg. time on site, and bounce rate)
\* A list of top pages
\* A list of top referrers
\* The ability to track more than one site
\* The ability to add more than one user to access the UI
While thereʼs a [13]public roadmap for new features and an [14]issue queue on GitHub (Fathom is open source, licensed with the MIT license), progress is slow but steady.

A few of the little things that I do actually miss from GA so far (which hopefully will show up in a future version of Fathom) include:
\* Browser / OS metrics (right now theyʼre not exposed in any way)
\* The ability to drill into referrer data (right now every siteʼs top referrer is "Google"... but you have no way of seeing the previous URL or search that led to the site)
\* The ability to see the graph in near-real time (though there is a nice continuously-updating "current visitors" metric)
All that said, Iʼm actually a lot happier with Fathom than I have been with Google Analytics, because the three metrics I care about most (time on site, visits, and referrers) are there, and ʼabove the foldʼ. With Google Analytics I rarely even visited the dashboard anymore because it felt like I had to spend 10 minutes getting all the data I needed—and I wasnʼt going to spend hours building custom dashboards for all my sites in different accounts!

Privacy

This was actually my main motivation behind actually making the move this year. Last year I deleted my Facebook account, and this year Iʼm trying to make sure my websites are even more divorced from ʼsocial trackingʼ. I have removed social integration from most of them (just sticking a ʼlikeʼ or ʼtweet thisʼ button means social giants get to track all users who hop through your site!), and my page size and overall page load times are so much the more better.

Not to mention, peopleʼs data and privacy are better-protected.

Google has traditionally been good (as far as I can tell) about not-being-evil, but in the past few years my faith (and many others, if the Hacker News crowd is to be believed) has really been shaken and much of the goodwill and trust developers and techies used to give Google has been lost—especially as they [15]sunset project after beloved project without much regard for the end users!

The main benefit of running Fathom on my own server, from a privacy perspective, is that every ounce of the userʼs tracking data is only stored, transmitted, and tracked by my own website—and I donʼt have to [16]agree to share that data with Google and Third Parties. I no longer have to sacrifice my siteʼs usersʼ privacy to Googleʼs data-hungry advertising AIs.

If I were to use the paid cloud/hosted version of Fathom instead of the self-hosted version, I can at least evaluate the tracking code myself, and see exactly what data is flowing from my siteʼs users into Fathomʼs systems. And judging by their past actions and words, I have a fair bit of trust that Fathomʼs founders will stick by their privacy-first stance.

Performance

Wondering how analytics trackers (most of which load asynchronously these days) affect overall page performance and bandwidth usage, I tested with both Google Analytics and Fathom to see how large the payload is and how fast the tracker javascript loaded, and whether it made any noticeable difference in overall page load time.

Google Analytics JS takes around 30-40ms to load (TTFB 20ms) on my home Internet connection, and after a test of loading a full static HTML page (comparable to jeffgeerling.com) a few hundred times with and without Google Analytics, the difference in full page load time was around 60ms. The JS that is returned totaled 17.2 KB gzipped (43.1 KB uncompressed).

Using Fathom running on a separate 1 GB VPS on Digital Ocean, I am able to load the tracker.js file in around 50ms (TTFB 45ms), and it affects the entire page load time similarly to the GA js, adding around 60ms total to render/processing time. The JS that is returned totaled 1.5 KB gzipped (2.7 KB uncompressed).
Analytics Tool tracker load time tracker size (gzipped)
Google Analytics 40ms 17.2 KB
Fathom 50ms 1.5 KB
Difference 1.25x slower 11x smaller

Now, if youʼre a content marketer whoʼs used to tossing a veritable soup of tracking systems into every website you work on (hey letʼs add Tealium! And Adobe Analytics! We need click heatmaps, so we should have HotJar too. Oh and Mixpanel is all the rage, so need that too...), then the difference of 15 KB might not mean much.

But Iʼm an engineer. I care about performance. I often browse the web on my phone in a waiting room, or a car on a road trip... and Internet is slow. Like dial-up slow. So if your website takes more than 30 seconds to load because you have over 1 MB in gzipped Javascript, Iʼm going to close out that window and go elsewhere.

I also ran the test using one of the most lightweight websites I maintain, a static site for my book, [17]Ansible for DevOps. Using Pingdom Tools a few times during the day to check before (with Google Analytics) and after (with Fathom), here are the results (averaged from three tests at a time each, spaced out by about 30 minutes, so 9 tests total for each analytics tool):
Tool Page Size Load time
Google Analytics 128.6 KB 367ms
Fathom 112.2 KB 226ms
Difference 16.4 KB smaller 1.6x faster

You should have a strategy for analytics, a budget for how much page weight and performance youʼre willing to sacrifice to get that sweet, sweet analytics data juice—most of the time the web developers just roll over when marketing or sales requires the addition of yet another analytics tool. But it shouldnʼt be so.

Performance of Fathom (server)

Thereʼs also the question of performance of Fathom, the open source Go application itself; Iʼm currently using it with an SQLite database (which gains about 300-500 KB/day so far), and the system has rarely been over 0.03 avg load, with fathom only consuming ~42 MB of RAM after a week of analytics.

If youʼre concerned you wouldnʼt be able to host it yourself, and maintain a decent backup, Fathom also offers paid plans—though theyʼre a bit expensive for my taste, especially if youʼre considering using it for some tiny sites that only get a few dozen or a few hundred pageviews a day.

Iʼm currently monitoring the Fathom server long-term with munin and will see how events like traffic spikes affect performance. Maybe this blog post will be the first test of that!

Summary

I set out to see if it would be painless to transition my personal web analytics from Google Analytics—the 800-pound gorilla of the web analytics world—to Fathom, an upstart which has yet to prove its mettle.

The experience has been extremely positive. The performance of Fathom on a 1 GB VPS, even while tracking 2 moderate-traffic and 4 light-traffic sites, has been much better than I expected. And even when that server is offline (I tested this, of course!), page load times and page weight have gotten better for every site Iʼve tested.

If you arenʼt required to track and store all the data that Google Analytics stores, and you can live without a few of the niceties, you should definitely consider using [18]Fathom!

References

Visible links
1. https://analytics.google.com/analytics/web/
3. https://www.w3counter.com/
4. https://statcounter.com/free-hit-counter/
6. https://en.wikipedia.org/wiki/Google_Analytics#Popularity
7. https://usefathom.com/
9. https://www.ansiblefordevops.com/
10. https://galaxy.ansible.com/geerlingguy/fathom
11. https://github.com/geerlingguy/ansible-role-fathom/blob/master/README.md
12. https://hub.docker.com/r/usefathom/fathom
13. https://trello.com/b/x2aBwH2J/fathom-roadmap
14. https://github.com/usefathom/fathom/issues
15. https://killedbygoogle.com/
16. https://www.google.com/analytics/terms/us.html
17. https://www.ansiblefordevops.com/
18. https://usefathom.com/

HackerNewsBot debug: Calculated post rank: 95 - Loop: 219 - Rank min: 80 - Author rank: 52

Moving from Go to PHP Again

Remember when I ditched Laravel for Golang?
Article word count: 712

HN Discussion: https://news.ycombinator.com/item?id=19111358
Posted by rakibtg (karma: 406)
Post stats: Points: 122 - Comments: 87 - 2019-02-08T03:48:34Z

\#HackerNews #again #from #moving #php

---

Article content:




Remember when [1]I ditched Laravel for Golang?

Well, after 2 years on Go, our shop applications are powered by PHP again.

Why?! You already said it was probably a bad business decision, and then you spend even more time on it?! Well, yeah, several reasons actually.

PHP improved a lot

PHP improved a lot during the last 3 years. It added [2]scalar argument type declarations, [3]return type declarations, [4]multi-catch exceptions, impressive [5]performance improvements and many more general improvements.

Symfony4 is a game changer

I’ve always been a big fan of [6]Symfony’s compatibility promise and their impressive 13-year track record proves they mean it.

So when [7]Symfony4 was released and I heard good things about it, I took it for a test drive by implementing a tiny part of our application in it.

Conclusion: it’s great. Really, really great.

A lot of effort went into simplifying the setup, making it a lot faster to bootstrap a Symfony application with much less work required configuring bundles. It’s now rivaling Laravel’s rapid development while at the same time encouraging decent development practices to ensure you don’t shoot yourself in the foot. And [8]it performs really well.

It was relatively easy to port our old Laravel application to Symfony, implement some new features the Go version of our application offered and undo some of the shortcuts I took earlier (most of them because of Laravel’s global helpers).

A nice side effect is that I’ve managed to substantially increase our test coverage in the process. Writing the same application in terms of functionality for a second third time really helps in that regard.

[9]Symfonyʼs debug bar

Symfony’s debug bar is an amazing tool. It shows you what happened during the journey from request to response, notifies you of warnings & deprecations and comes with a built-in profiler that you can easily hook into to benchmark parts of your own code.

[10]Symfonyʼs profiler

After learning [11]Symfony’s Form component, I’d rather not go without it again. It makes it trivial to render an accessible form that can be re-used in several places, validating the form upon submit and then populating a PHP object from the form data safely.

$user = $this->getUser();
$form = $this->createForm(UserBillingInfoType::class, $user) ->handleRequest($request); if ($form->isSubmitted() && $form->isValid()) { // $user is already populated with the form values at this point
// itʼs valid, so we can update the database and redirect the user now
}

Doctrine is another piece of software that really improved our overall application. Your models ([12]entities) are normal PHP classes and relations ([13]associations) are normal references, making it easy to test your domain logic without having to worry about the database implementation.

$user = new User();
$user->addLicense(new License());
$manager->persist($user); // both user and its license will be saved

In Doctrine all operations are wrapped in a SQL transaction by default. That’s a big plus for me as it guarantees atomicity, which involved more work to get right in Eloquent.

Go is (still) great

Honestly, Go is great. Its simplicity is refreshing and you can’t get anywhere near that kind of performance using PHP ^1. I would still pick it if we need a small API or something that requires high throughput.

Our shops however are more monolithic with a lot of server-side rendering. While that’s certainly doable in Go (as the last 2 years proved), it’s more maintainable for us to do it in PHP right now.

Side note: without the experience gained from our years on Go, I probably wouldnʼt have started [14]Fathom Analytics. So perhaps it wasnʼt such a bad business decision after all?

Making the correct business decision

One reason not mentioned so far is that over the last year or so, I’ve been approached by several companies interested to take over one of our products.

They were a little surprised to hear our stack involved Golang and some flat out told us they’d prefer PHP, because that’s what most of our products ([15]mc4wp.com, [16]boxzillaplugin.com and [17]htmlforms.io) rely upon. And I don’t blame them.

^1 Just for fun, I compared apples and oranges again by benchmarking the login page (which doesnʼt hit any database) for both application versions using [18]Siege.

The Symfony application (PHP 7.3, OPcache enabled, optimized autoloader) handles about 1470 req/s. The Go application (compiled using Go v1.11) averages about 18600 req/s.

References

Visible links
1. https://dannyvankooten.com/laravel-to-golang/
2. https://secure.php.net/manual/en/functions.arguments.php#functions.arguments.type-declaration
3. https://secure.php.net/manual/en/functions.returning-values.php#functions.returning-values.type-declaration
4. https://wiki.php.net/rfc/multiple-catch
5. http://www.zend.com/en/resources/php7_infographic
6. https://symfony.com/doc/current/contributing/code/bc.html
7. https://symfony.com/4
8. http://www.phpbenchmarks.com/en/
11. https://symfony.com/doc/current/forms.html
12. https://www.doctrine-project.org/projects/doctrine-orm/en/latest/reference/basic-mapping.html#basic-mapping
13. https://www.doctrine-project.org/projects/doctrine-orm/en/latest/reference/working-with-associations.html#working-with-associations
14. https://usefathom.com/
15. https://mc4wp.com/
16. https://boxzillaplugin.com/
17. https://htmlforms.io/
18. https://www.joedog.org/siege-home/

HackerNewsBot debug: Calculated post rank: 110 - Loop: 107 - Rank min: 100 - Author rank: 36

If Software Is Funded from a Public Source, Its Code Should Be Open Source

If we pay for it, we should be able to use it. Perhaps because many free software coders have been outsiders and rebels, less attention is paid to the use of open source in government departments than…
Article word count: 1088

HN Discussion: https://news.ycombinator.com/item?id=19077913
Posted by jrepinc (karma: 1304)
Post stats: Points: 200 - Comments: 68 - 2019-02-04T16:55:50Z

\#HackerNews #code #from #funded #its #open #public #should #software #source

---

Article content:

If we pay for it, we should be able to use it.

Perhaps because many free software coders have been outsiders and rebels, less attention is paid to the use of open source in government departments than in other contexts. But itʼs an important battleground, not least because there are special dynamics at play and lots of good reasons to require open-source software. Itʼs unfortunate that the most famous attempt to convert a government IT system from proprietary code to open source—the city of Munich—proved such a difficult experience. Although last year saw [1]a decision to move back to Windows, that seems to be more a failure of IT management, than of the code itself. Moreover, itʼs worth remembering that the Munich project began back in 2003, when it was a trailblazer. Today, there are [2]dozens of large-scale migrations, as TechRepublic reports:
Most notable is perhaps the French Gendarmerie, the countryʼs police force, which has switched 70,000 PCs to Gendbuntu, a custom version of the Linux-based OS Ubuntu. In the same country 15 French ministries have made the switch to using LibreOffice, as has the Dutch Ministry of Defence, while the Italian Ministry of Defence will switch more than 100,000 desktops from Microsoft Office to LibreOffice by 2020 and 25,000 PCs at hospitals in Copenhagen will move from Office to LibreOffice.
More are coming through all the time. The Municipality of Tirana, the biggest in Albania, has just announced it is [3]moving thousands of desktops to LibreOffice, and nearly [4]80% of the city of Barcelonaʼs IT investment this year will be in open source.

One factor driving this uptake by innovative government departments is the potential to cut costs by avoiding constant upgrade fees. But itʼs important not to overstate the "free as in beer" element here. All major software projects have associated costs of implementation and support. Departments choosing free software simply because they believe it will save lots of money in obvious ways are likely to be disappointed, and that will be bad for open sourceʼs reputation and future projects.

Arguably as important as any cost savings is the use of open standards. This ensures that there is no lock-in to a proprietary solution, and it makes the long-term access and preservation of files much easier. For governments with a broader responsibility to society than simply saving money, that should be a key consideration, even if it hasnʼt been in the past.

Open-source advocates have rightly noted that free software is a natural fit for any organization that requires solutions based on open standards, interoperability and re-usable components—key elements of the European Commissionʼs new [5]digital strategy, for example. One of the leaders here is the UK government. In 2014, it announced a new policy of [6]"Making things open, making things better". It achieved this by setting [7]Open Document Format for Office Applications Version 1.2 as the [8]default format for sharing or collaborating with UK government documents. Itʼs produced an interesting [9]review of how things have gone in the last four years, which concludes:
We cannot have important documents published in formats which do not meet open standards. Government documents are for everyone. Whether youʼre using Windows, Mac, GNU/Linux, Chrome OS, iOS, Android, or any other system—you have the right to read what we have written and we will continue on our journey to make documents open and accessible.
The use of open standards is not the only big benefit of moving to open source. Another is transparency. Recently it emerged that [10]Microsoft has been gathering personal information from 300,000 government users of Microsoft Office ProPlus in the Netherlands, without permission and without documentation:
Microsoft systematically collects data on a large scale about the individual use of Word, Excel, PowerPoint and Outlook. Covertly, without informing people. Microsoft does not offer any choice with regard to the amount of data, or possibility to switch off the collection, or ability to see what data are collected, because the data stream is encoded. Similar to this practice in Windows 10, Microsoft has included separate software in the Office software that regularly sends telemetry data to its own servers in the United States.
Moving to open-source solutions does not guarantee that personal data will not leak out, but it does ensure that the problems, once found, can be fixed quickly by government IT departments—something that isnʼt the case for closed-source products. This is a powerful reason why public funds should mean open source—or as a site created by the Free Software Foundation Europe puts it: [11]"If it is public money, it should be public code as well".

The site points out some compelling reasons why any government code produced with public money should be free software. They will all be familiar enough to readers of Linux Journal. For example, publicly funded code that is released as open source can be used by different departments, and even different governments, to solve similar problems. That opens the way for feedback and collaboration, producing better code and faster innovation. And open-source code is automatically available to the people who paid for it—members of the public. They too might be able to offer suggestions for improvement, [12]find bugs or build on it to produce exciting new applications. None of these is possible if government code is kept locked up by companies that write it on behalf of taxpayers.

Once again, the natural fit of open source with public computing is evident. Indeed, when you think about it, it seems ridiculous that public money would be used to produce anything but public code. The Basque Country understood that back in 2012 and brought in a law that required all [13]software developed for the government there should be released as open source. More recently, the Canadian government has made the connection too. Its new [14]Directive on Management of Information Technology says:
Where possible, use open standards and open source software first.

...

If a custom-built application is the appropriate option, by default any source code written by the government must be released in an open format via Government of Canada websites and services designated by the Treasury Board of Canada Secretariat.

All source code must be released under an appropriate open source software license.
The fact that this approach is not already the norm is something of a failure on the part of the Free Software community. Perhaps itʼs time to drop the snobbery about open source in government and put more effort into turning it into the next huge win for the world of free software.

References

Visible links
1. https://www.techrepublic.com/article/end-of-an-open-source-era-linux-pioneer-munich-confirms-switch-to-windows-10
2. https://www.techrepublic.com/article/linux-to-windows-10-why-did-munich-switch-and-why-does-it-matter
3. https://blog.documentfoundation.org/blog/2018/11/22/municipality-of-tirana
4. https://joinup.ec.europa.eu/news/growing-100-2020
5. https://ec.europa.eu/info/sites/info/files/strategy/decision-making_process/documents/ec_digitalstrategy_en.pdf
6. https://gds.blog.gov.uk/2014/07/22/making-things-open-making-things-better
7. http://docs.oasis-open.org/office/v1.2/OpenDocument-v1.2.html
8. https://www.gov.uk/government/publications/open-standards-for-government/sharing-or-collaborating-with-government-documents
9. https://gdstechnology.blog.gov.uk/2018/04/27/open-document-format-in-government-an-update
10. https://www.privacycompany.eu/en/impact-assessment-shows-privacy-risks-microsoft-office-proplus-enterprise
11. https://publiccode.eu/
12. https://juliareda.eu/fossa/#bugbounty
13. https://joinup.ec.europa.eu/news/basque-countrys-open-source
14. https://www.tbs-sct.gc.ca/pol/doc-eng.aspx?id=15249

HackerNewsBot debug: Calculated post rank: 156 - Loop: 164 - Rank min: 100 - Author rank: 54

Police stop people for covering their faces from facial recognition camera

Metropolitan Police had said people declining to be scanned would 'not necessarily be viewed as suspicious'
Article word count: 2956

HN Discussion: https://news.ycombinator.com/item?id=19068452
Posted by walterbell (karma: 36750)
Post stats: Points: 78 - Comments: 62 - 2019-02-03T08:31:52Z

\#HackerNews #camera #covering #faces #facial #for #from #people #police #recognition #stop #their

---

Article content:




A man has been fined after refusing to be scanned by controversial [1]facial recognition cameras being trialled by the [2]Metropolitan Police.

The force had put out a statement saying “anyone who declines to be scanned will not necessarily be viewed as suspicious”. However, witnesses said several people were stopped after covering their faces or pulling up hoods.

Campaign group Big Brother Watch said one man had seen placards warning members of the public that automatic facial recognition cameras were filming them from a parked police van.

“He simply pulled up the top of his jumper over the bottom of his face, put his head down and walked past,” said director Silkie Carlo.

“There was nothing suspicious about him at all … you have the right to avoid [the cameras], you have the right to cover your face. I think he was exercising his rights.”

Ms Carlo, who was monitoring Thursday’s trial in Romford, London, told The Independent she saw a plainclothed police officer follow the man before a group of officers “pulled him over to one side”.

She said they demanded to see the man’s identification, which he gave them, and became “accusatory and aggressive”.

“The guy told them to p*** off and then they gave him the £90 public order fine for swearing,” Ms Carlo added. “He was really angry.”

A spokesperson said officers were instructed to “use their judgment” on whether to stop people who avoid cameras.

Created with Sketch. Created with Sketch.

Dog walkers in Aston Rowant National Nature Reserve on the Chilterns escarpment, in Oxfordshire. Snowfall and icy conditions are expected to cause widespread travel disruption after temperatures plummeted as low as minus 15.4C (4.3F) overnight.
A person walks past the frozen Sefton Park Lake in Liverpool after the UK had its coldest night of the winter so far as the cold snap continues to cause icy conditions across the country
Police searching near the scene in Islington, north London where 17-year-old boy, Nedim Bilgin, was pronounced dead after being stabbed on Tuesday evening
A vehicle navigates in snowy conditions near Newby Head in North Yorkshire, as up to 10cm of snow could fall on higher ground as temperatures drop across the UK this week
A Union flag flies from a pole in front of the Elizabeth Tower, commonly known as Big Ben, near the Houses of Parliament in central London. Despite the humiliating rejection of Prime Minister Theresa Mayʼs Brexit deal, Britain is no closer to knowing the end result of its vote to leave the European Union. A raft of amendments to be voted on by MPs on Tuesday threaten to further muddy the waters as the clock ticks down to Britainʼs scheduled departure from the EU on March 29
Irish foreign minister Simon Coveney being interviewed by host Andrew Marr on the BBC1 current affairs programme, Simon Coveney told the Show that the EU would not ratify a Brexit deal without a backstop.
Michael Russell, the Scottish governmentʼs minister with responsibility for Brexit negotiations attends a Rally for Europe at Augustine United Church in Edinburgh, Scotland. With only two months to go until the UK is scheduled to leave the EU, people attended the rally to listen to speakers as they called for Brexit to be halted and for a second people’s vote to take place.
Former engineers Derek Mack (left) and Mike Kelloway, both from the Isle of Wight, inside the remains of the Black Arrow projectile, the UKʼs only rocket to successfully launch a satellite into orbit, are unveiled at a storage facility in Penicuik, near Edinburgh, after almost 50 years languishing in the Australian Outback
Former Scottish first minister Alex Salmond makes a statement outside Edinburgh Sheriff Court after he was arrested and charged by police. No further details of the charge against the 64-year-old have been released. A Police Scotland spokeswoman said: "We can confirm that a 64-year-old man has been arrested and charged, and a report will be sent to the procurator fiscal
A couple walk through a park, in High Wycombe, as some areas of the UK see the first snow of the year
Ttributes left outside Cardiff City Stadium after a plane carrying their new signing Emiliano Sala disappeared over the English Channel on Monday night. The aircraft disappeared off radar after losing contact with air traffic control around 8:20pm, with rescue attempts abandoned through the night due to worsening conditions. Lifeboats and helicopters resumed the search 8am on Tuesday morning, but the latest update from Guernsey Police is that there is “no trace of the aircraft” and officials have now said that they are "not expecting anyone to be alive"
A super blood wolf moon over the peace statue on Brighton seafront during a lunar eclipse
Forensic officers inspect the remains of the van used as a car bomb on an attack outside Derry Court House in Londonderry, Northern Ireland. Dissident republicans are suspected to have carried out the attack which has been condemned by Northern Ireland politicians
Womenʼs march for “bread and roses”, rally against austerity in London
A team of specialists who are tasked with piecing together the iconic dinosaur Dippy unpack bespoke crates containing the bone structure at Kelvingrove Art Gallery and Museum in Glasgow. As part of a road trip across the UK, Dippy has arrived in Scotland and will be on show
A farmer rides a quad bike as snow and frost settles on the peaks of the hills in the Brecon Beacons
Theresa May speaking during Prime Ministerʼs Questions in the House of Commons the day after her Brexit deal was defeated in Parliament
Police monitor Brexit protesters outside the houses of parliament in London. Parliamentarians are voting on the postponed Brexit EU Withdrawal Agreement, commonly known as The Meaningful Vote, deciding on Britainʼs future relationship with the European Union
Andy Murray produced one of the performances of his career, but after more than four hours and with an injured hip that will inevitably end his career at some point this year, he exited the Australian Open in the first round at the hands of 22nd seed Roberto Bautista Agut. It looked to be a straight-forward victory for the Spaniard as he surged into a two-set lead, but Murray is not a three-time Grand Slam champion for nothing and he supremely levelled the match despite hobbling around the court from the get-go. However, in the end his hip injury just took too much out of him, and Bautista Agut clinched a breath-taking 6-4, 6-4, 6-7 (5-7), 6-7 (4-7), 2-6 victory to reach the second round where heʼll face Australiaʼs John Millman
British Prime Minister, Theresa May and her husband Philip attend Sunday morning prayers at her local church in Maidenhead, Britain, 13 January 2019. Members of the British Parliament are due to vote on Theresa Mayʼs Brexit deal on 15 January 2019, with Britain set to leave the European Union on 29 March 2019.
Demonstrators wearing yellow vests protest near Broadcasting House on in London, England. Demonstrators from both the left and the right wings of British politics have adopted the ʼGilets Jaunesʼ or Yellow Vest form of protest that first became prominent in France throughout November and December 2018. Various groups meet in London today to protest the Brexit deal due to be voted on in the UK Parliament
An emotional Andy Murray revealed this yearʼs Australian Open could be the last tournament of his career. Struggling to hold back tears, the 31-year-old Scot said that the continuing pain from what he described as his “severely damaged right hip” had led him to decide to end his career this year. He said that he had been planning to make this summer’s Wimbledon his farewell tournament, but feared that he might have to bring down the curtain before then
Britainʼs Prime Minister Theresa May and Japanʼs Prime Minister Shinzo Abe were greeted by Chase Bridge Primary School children waving UK and Japanese national flags during a visit to Twickenham Rugby Stadium in, west London. Abe is visiting Britain and set to hold talks with May following her visit to Tokyo and Kyoto last year
Former leader of the Liberal Democrats, Tim Farron (right) speaks to a pro-Brexit protester outside the Houses of Parliament. MPs will resume debating Theresa Mayʼs Brexit plan a month after she postponed the original commons vote. Met police have been urged to take action against protesters who abuse MPʼs. This comes after Conservative MP Anna Soubry was verbally abused and called ʼNazi scumʼ by pro-Brexit protesters on 7 January
Alex Salmond speaking outside the Court of Session in Edinburgh after it ruled that the Scottish Government acted unlawfully regarding sexual harassment complaints against the former first minister
Britainʼs Prime Minister Theresa May meets with a mother and baby during a visit to a facility within Alder Hey Childrenʼs Hospital in Liverpool. May made a speech setting out the governmentʼs long-term plan for the National Health Service
The Lions part group of professional performers on Bankside, during the annual Twelfth Night celebrations on the South Bank in central London.
Elliott List celebrates after scoring Gillinghamʼs first goal during the FA Cup third round match between against Cardiff City at Priestfield Stadium. The League One side went on to knock the Premier League out of the competition
Frances Connolly, 52, and Patrick Connolly, 54, from Moira, County Armagh in Northern Ireland, who scooped a £115 million EuroMillions jackpot in the New Yearʼs Day lottery draw, during a photocall at the Culloden Estate and Spa in Holywood, Belfast, as they announce their win
The offshore patrol vessel HMS Mersey leaves Portsmouth Harbour as it is deployed to patrol the English channel following an increase in migrants crossing in small boats
Rail protestors wearing masks with the faces of Prime Minister Theresa May and Transport Secretary Chris Grayling outside Kings Cross St Pancras station in London ahead of a visit by Labour leader Jeremy Corbyn to highlight ʼrising rail fares and falling standards of service on Britainʼs railwaysʼ as part of Labourʼs Rail Action Day
Police restraining a man after he stabbed three people at Victoria Station in Manchester last night. Two commuters - a man and woman in their 50s - were taken to hospital with knife injuries and a British Transport Police (BTP) officer was stabbed in the shoulder. Police said a man was arrested on suspicion of attempted murder and remains in custody
Competitors in fancy dress run across the Pennine tops near Haworth, West Yorkshire, in the annual Auld Lang Syne Fell race which attracts hundreds of runners every year
UK Border Force Cutter, ʼSearchʼ berths at Ramsgate Harbour. The growing number of migrants attempting to cross the English Channel has been declared a "major incident" by UK home secretary Sajid Javid
People remove a Santa hat on the 20 metres tall steel sculpture ʼAngel of the Northʼ, outside Gateshead, Tyne and Wear. The pranksters responsible climbed up overnight to remove it. One of the group, dressed as Dr Seuss character the Grinch, was photographed pulling the hat down from the 65ft structure, with the help of three others dressed as Santa. The hat had been placed on top of the famous Anthony Gormley artwork on Christmas Eve, prompting delight from onlookers
The steam locomotive U Class 31806 makes its way past Corfe Castle during the Winter Warm Up on the Swanage Railway in Dorset
The scene in Launcelot Close, Andover, where the body of a man has been found after an explosion caused a building to collapse
Shoppers enter the Selfridges store on Oxford Street during the Boxing Day sales in London
Barbary macaques at Blair Drummond Safari Park have a great time finding the tasty morsels hidden amongst the wrapped parcels given to them by keepers. The animals need care and attention every day and Christmas is no different when the animals are checked, fed and given enrichment to keep them stimulated
A Santa hat on the 20 metres tall steel sculpture ʼAngel of the Northʼ, outside Gateshead, Tyne and Wear. The hat is believed to have been added overnight, with passers-by stopping to take photos in the early hours of the morning. The Sir Antony Gormley sculpture was installed in 1998, with celebrations marking its 20th birthday this year
Team Dash and Splash swimmers, some in fancy dress, getting into the Irish Sea at Bangor beach in Northern Ireland
Members of the Shakti Sings choir sing as druids, pagans and revellers gather in the centre of Stonehenge, hoping to see the sun rise, as they take part in a winter solstice ceremony at the ancient neolithic monument near Amesbury. The winter solstice is the shortest day of the year and the event is claimed to be more important in the pagan calendar than the summer solstice, because it marks the ʼre-birthʼ of the sun for the new year
Environmental protestors from the Extinction Rebellion group chant during a demonstration outside BBC Broadcasting House in London. The group was calling for the media organisation to provide further in-depth coverage of climate-related issues in future reporting
A police helicopter flies over Gatwick airport in search of the person operating the drone that has caused the airport to be closed today
There was controversy in the Commons today as Jeremy Corbyn allegedly called the Prime Minister a stupid woman under his breath after she made a joke at PMQs
Jose Mourinho after he was sacked by Manchester United with immediate effect. The Portuguese leaves United sixth in the table with the 3-1 defeat to Liverpool at Anfield on Sunday proving his final game in charge. A club statement read: "Manchester United announces that manager Jose Mourinho has left the club with immediate effect. The club would like to thank Jose for his work during his time at Manchester United and to wish him success in the future. A new caretaker manager will be appointed until the end of the current season, while the club conducts a thorough recruitment process for a new, full-time manager."
Firefighters and police officers attend a memorial service at the Harrods Bombing memorial in west London, on the 35th anniversary of the terrorist attack which left three police officers and three members of the public dead, on December 17, 1983
Theresa May has hit out at Mr Blair, accusing him of "insulting"the British people and the office of prime minister by "undermining" Brexit talks with calls in Brussels for a second referendum.
Chester Zoo after a fire broke out in the Monsoon Forest habitat area.
Fracking has been halted at the Preston Road site in Lancashire after a series of tremors peaking at 0.9 magnitude
Irish Taoiseach Leo Varadkar today told Theresa May that he expects assurances that there will be no hard border between Northern Ireland and the Republic of Ireland

"Officers stopped a man who was seen acting suspiciously in Romford town centre during the deployment of the live facial recognition technology," a statement said.

"After being stopped the man became aggressive and made threats towards officers. He was issued with a penalty notice for disorder as a result."

Eight people were arrested during the eight-hour trial, although only three were a direct result of facial recognition technology.

A 15-year-old boy identified by the cameras was arrested on suspicion of robbery but released with no further action.

A 28-year-old man was arrested on suspicion of false imprisonment and another man, 35-year-old man, was arrested on suspicion of breach of a molestation order.

The other arrests were two teenage boys accused of robbery, a 17-year-old boy accused of firing a gun and two men, aged 25 and 46, for drug possession.

The deployment trial was due to continue on Friday, but rescheduled because of forecast snow and cold temperatures causing "low footfall".

Monitors saw several other people stopped outside Romford station, in north east London, including a student who had pulled his hood up and a man handcuffed and put in a police van.

Activists from the Liberty human rights group said they spoke to a youth worker who was stopped because he “looked like someone” on a watchlist, but had been misidentified.

Scotland Yard said the two-day deployment of cameras in Romford would be the last of 10 trials of the controversial technology.

Facial recognition trial in London’s West End

The Independent revealed that [3]more than £200,000 was spent on six deployments that resulted in no arrests between August 2016 and July last year. Two people wanted for violent offences were arrested after a trial in December.

Critics have called the force’s use of facial recognition a “shambles” and accused Scotland Yard of wasting public money.

Automatic facial recognition software compares live footage of people’s faces to photos from a watchlist of selected images from a police database.

Any potential matches are flashed up as an alert to officers, who then compare the faces and decide whether to stop someone.

The Metropolitan Police has described the deployments as “overt” and said members of the public were informed facial recognition was being used by posters and leaflets.

But no one questioned by The Independent after they [4]passed through a scanning zone in central London in December had seen police publicity material, and campaigners claim the technology is being rolled out “by stealth”.

Detective Chief Superintendent Ivan Balhatchet, Scotland Yard’s lead for facial recognition, said a full independent evaluation will be carried out.

“The technology used in Romford forms part of the Metʼs ongoing efforts to reduce crime in the area, with a specific focus on tackling violence," he added.

“As with all previous deployments the technology was used overtly. We continue to engage with many different stakeholders, some who actively challenge our use of this technology."

References

Visible links
1. https://www.independent.co.uk/topic/facial-recognition
2. https://www.independent.co.uk/topic/metropolitan-police
3. https://www.independent.co.uk/news/uk/home-news/facial-recognition-uk-police-met-arrests-london-cost-false-positives-accuracy-a8723756.html
4. https://www.independent.co.uk/news/uk/home-news/facial-recognition-cameras-london-met-police-suspects-arrests-identity-a8687481.html

HackerNewsBot debug: Calculated post rank: 72 - Loop: 168 - Rank min: 60 - Author rank: 49

Something mysterious is blocking car key fobs from working in an Alberta town

Key fobs that suddenly won't unlock vehicles. Cars that won't start. Alarms that go off for no reason. Something mysterious is thwarting drivers outside a grocery store in the small Alberta town of…
Article word count: 580

HN Discussion: https://news.ycombinator.com/item?id=19046475
Posted by drpgq (karma: 1089)
Post stats: Points: 144 - Comments: 135 - 2019-01-31T17:17:16Z

\#HackerNews #alberta #blocking #car #fobs #from #key #mysterious #something #town #working

---

Article content:



Key fobs that suddenly wonʼt unlock vehicles. Cars that wonʼt start. Alarms that go off for no reason and canʼt be quieted. Something mysterious is thwarting drivers outside a grocery store in the small Alberta town of Carstairs — and itʼs sparking all kinds of theories.

The problems have been happening for weeks in the parking lot outside the Westview Co-op grocery store in Carstairs, a town of about 4,000 about 60 kilometres north of Calgary.

A longtime employee at the dollar store right across the street from the Co-op says itʼs all she hears some folks talk about when they come into her store to buy a battery for their fobs — and then discover that doesnʼt solve the problem.

"Iʼve been at the dollar store almost four years," Laura Strate said, with a laugh.

"Itʼs just bizarre. People are actually scared to go to the Co-op now because they donʼt know if their cars are going to start."

ʼPeople are scared to go to the Co-opʼ

Itʼs lighting up social media, Strate says.

"People start getting on there and saying, ʼOK, Iʼve been to the Co-op. Is anyone else having this problem?ʼ And all of a sudden we have 160 people commenting, ʼYes, that happened to me last week,ʼ or ʼThat happened to me today,ʼ" Strate said.

"I would say itʼs been probably a good month where people have known there is a problem and itʼs not just someone that says, ʼOh, my car doesnʼt work at Co-op.ʼ"

Puzzled store calls in electricians — and even federal ministry

Westview Co-operative Association Carstairs says itʼs taking the matter very seriously.

"I think it is frustrating," Stephen Kennedy, the storeʼs asset protection manager, told CBC News.

"To see the level of frustration for our team and our guests is where our concern is. Thatʼs why we are taking extra steps to ensure we are driving the solution."

Electricians from J. Williams Electric in Carstairs rode to the rescue — but the mystery endured.

"We have shut down the power source in our store this past Monday night to determine that the source of the interference isnʼt emanating from our location," Kennedy said.

"The location is unknown so weʼre just waiting for followup from the ministry."

He said theyʼre working with a team to try and figure out whatʼs going on, including a federal government agency.

"We have partnered with neighbours, stakeholders and the Ministry of Innovation, Science and Economic Development."

The Didsbury RCMP say theyʼve heard the rumours and had reports so are looking into it — but itʼs not a priority for them.

ʼPut your tin foil hats onʼ

Meanwhile Strate says people are coming up with their own theories for whatʼs affecting the fobs.

"We joke around here in Carstairs, ʼYeah, put your tin foil hats on,ʼ" Strate said.

People have speculated it could be a security system recently installed by the food store, a nearby Tim Hortons that opened just after Christmas, ongoing highway construction or even the demolition of an old Rona hardware building.

But they are all just theories.

"It is pretty bizarre," Strate said.

CBC News reached out to the Ministry of Innovation, Science and Economic Development for comment, but hadnʼt received a response by publication time.

A mystery is unfolding in Carstairs, and its become the talk of the town. It seems key fobs for some cars have stopped working in the parking lot of a local grocery store. People in town are trying to figure out why its happening. The CBCʼs David Bell is trying to find some answers too, and joins Doug now in studio 3:30

HackerNewsBot debug: Calculated post rank: 141 - Loop: 339 - Rank min: 100 - Author rank: 22

Apple blocks Google from running its internal iOS apps

Google joins Facebook in Apple’s banning spree
Article word count: 204

HN Discussion: https://news.ycombinator.com/item?id=19048976
Posted by rising-sky (karma: 903)
Post stats: Points: 236 - Comments: 130 - 2019-01-31T21:20:26Z

\#HackerNews #apple #apps #blocks #from #google #internal #ios #its #running

---

Article content:




Illustration by Alex Castro / The Verge

Apple has now shut down Google’s ability to distribute its internal iOS apps, following a similar shutdown that was [1]issued to Facebook earlier this week. A person familiar with the situation tells The Verge that early versions of Google Maps, Hangouts, Gmail, and other pre-release beta apps have stopped working today, alongside employee-only apps like a Gbus app for transportation and Google’s internal cafe app.

We’ve reached out to both Apple and Google to comment on the situation, but neither company responded by publish time.

Apple’s move to block Google’s developer certificate comes just a day after [2]Google disabled its Screenwise Meter app following press coverage. Google’s private app was designed to monitor how people use their iPhones, similar to Facebook’s research app. Google’s app also relied on Apple’s enterprise program, which enables the distribution of internal apps within a company.

In an earlier statement over Facebook’s certificate removal, Apple did warn that “any developer using their enterprise certificates to distribute apps to consumers will have their certificates revoked.” Apple is clearly sticking to its rules and applying them equally to Facebook, Google, and likely many other companies that get caught breaking Apple’s rules in the future.

References

Visible links
1. https://www.theverge.com/2019/1/30/18203551/apple-facebook-blocked-internal-ios-apps
2. https://www.theverge.com/2019/1/30/18204064/apple-google-monitoring-phone-usage-screenwise-meter

HackerNewsBot debug: Calculated post rank: 200 - Loop: 53 - Rank min: 100 - Author rank: 74

Apple blocks Facebook from running its internal iOS apps

Facebook’s internal iOS apps simply don’t launch anymore.
Article word count: 428

HN Discussion: https://news.ycombinator.com/item?id=19035834
Posted by epaga (karma: 6531)
Post stats: Points: 291 - Comments: 106 - 2019-01-30T15:32:40Z

\#HackerNews #apple #apps #blocks #facebook #from #internal #ios #its #running

---

Article content:




Illustration by Alex Castro / Th

Apple has shut down Facebook’s ability to distribute internal iOS apps, from early releases of the Facebook app to basic tools like a lunch menu. A person familiar with the situation tells The Verge that early versions of Facebook, Instagram, Messenger, and other pre-release “dogfood” (beta) apps have stopped working, as have other employee apps, like one for transportation. Facebook is treating this as a critical problem internally, we’re told, as the affected apps simply don’t launch on employees’ phones anymore.

The shutdown comes in response to news that Facebook has been using Apple’s program for internal app distribution to track teenage customers with a “research” app.

"“A clear breach of [Facebook’s] agreement with Apple”"

That app, [1]revealed yesterday by TechCrunch, was distributed outside of the App Store using Apple’s enterprise program, which allows developers to use special certificates to install more powerful apps onto iPhones. Those apps are only supposed to be used by a company’s employees, however, and Facebook had been distributing its tracking app to customers. Facebook later said [2]it would shut down the app.

This poses a huge issue for Facebook. While Apple provides other tools a company can use to install apps internally, Apple’s enterprise program is the main solution for widely distributing internal apps and services. We’ve reached out to Facebook for comment.

In a statement [3]given to Recode, Apple said that Facebook was in “clear breach of their agreement with Apple.” Any developer that breaches that agreement, Apple said, has their distribution certificates revoked, “which is what we did in this case to protect our users and their data.” We’ve reached out to Apple for comment on shutting down Facebook’s other internal apps.

Revoking a certificate not only stops apps from being distributed on iOS, but it also stops apps from working. And because internal apps by the same organization or developer may be connected to a single certificate, it can lead to immense headaches like the one Facebook now finds itself in where a multitude of internal apps have been shut down.

Apple and Facebook have already been bickering over privacy, but this is the first instance of Apple taking an action that directly shuts down some of Facebook’s activities. Last March, Apple CEO Tim Cook [4]criticized Facebook’s handling of the Cambridge Analytica data sharing scandal, saying, “I wouldn’t be in this situation” if he were running the company. Facebook CEO Mark Zuckerberg [5]later said the comments were “extremely glib” and spoke of Apple as a company that “work [s]hard to charge you more.”

References

Visible links
1. https://techcrunch.com/2019/01/29/facebook-project-atlas/
2. https://www.theverge.com/facebook/2019/1/30/18203349/facebook-research-app-apple-shutdown
3. https://www.recode.net/2019/1/30/18203231/apple-says-its-banning-facebooks-research-that-collected-users-personal-information
4. https://www.recode.net/2018/3/28/17172212/apple-facebook-revolution-tim-cook-interview-privacy-data-mark-zuckerberg
5. https://www.theverge.com/2018/4/2/17188660/mark-zuckerberg-tim-cook-comments-facebook-extremely-glib

HackerNewsBot debug: Calculated post rank: 229 - Loop: 115 - Rank min: 100 - Author rank: 105

From “Hello World” to VP of Engineering at Reddit (2017)

My Career Journey, Part 1
Article word count: 3438

HN Discussion: https://news.ycombinator.com/item?id=19022177
Posted by karim (karma: 731)
Post stats: Points: 132 - Comments: 54 - 2019-01-28T23:32:22Z

\#HackerNews #2017 #engineering #from #hello #reddit #world

---

Article content:




[1]Go to the profile of Nick Caldwell

Hello World! The following is transcribed from an April, 2017 speech I gave about my career journey and leadership principles.

My 2nd day @ Reddit

Hi, I’m Nick Caldwell, and I’m the VP of Engineering at Reddit, this country’s 4th most popular website and 1st most popular source for funny cat pictures. Before working at Reddit, I spent 13 years at Microsoft where I worked my way up from a position as software engineering intern to become partner-level General Manager of a 300 person organization called Power BI.

I’ve been invited here to talk about my leadership journey, my thoughts on diversity and inclusion, and what’s going on at Reddit. You should know that I get really embarrassed talking about myself in public. But, as one of the few African American tech execs out here in Silicon Valley, I hope my story will help inspire you to do better at work or get where you want to go in life.

Now, I’m going to talk for about an hour, and if that’s too long I totally understand. I don’t have slides or anything flashy, I’m just going to talk. If you get bored, my only ask is that instead of playing Mario Run or doing actual work, you pull out your phones and get the latest Reddit app. We just launched an April Fool’s Day project called [2]r/place which will keep you distracted for the remainder of the time.

Where to begin? If I’m going to tell you about my leadership journey then I’ve got to start with what inspired me and what motivated me early on.

I was born in Alexandria, Virginia to a white mother who carried me to term out of wedlock, against the objections of her own family, and gave me up for adoption immediately after I was born. This sounds bad but, trust me, everything turns out okay in the end (at least I hope it does).

A few weeks later I was adopted by my mother, a black woman and first generation Jamaican immigrant who had recently finished her college degree in early childhood education and home economics. She was married to a black Vietnam veteran who, after a short career as a Navy JAG officer, later went on to become a public defender, advocate for social justice, educator, and jazz band manager.

These two wondrous people, fully able to have children and living solidly in the lower-middle class, decided to adopt me. And although I grew up with relatively little, and their relationship would strain over the years, they poured everything they had into educating and motivating me. While the act of having and raising children is always about putting someone else’s life before your own, to this day I am in absolute awe of my parent’s decision and see it as a continual, ongoing, every-moment reminder to give back. It is also a reminder of how much in life depends on chance.

But enough of that and back to inspiration. My early life was defined by 3 motivations:

First, even at a young age my father (who if you couldn’t tell from my earlier description was into a lot of random things) was continually exposing me to new topics: jazz, classical, and R&B music, chess, calligraphy, all sorts of book, chemistry sets, etc. One day he brought home from work a Tandy 1000 computer to replace his typewriter. This was one of the very first personal computers if you didn’t know. It had a 086 processor and had been upgraded to a whopping 640 kilobytes of memory running MS-DOS 2.0. Basically a pocket calculator in today’s terms. Yet, I was immediately fascinated by this thing, mostly because of the video games it came with but also because it quickly became how I spent the majority of time with my father. I’d sit on his lap for hours and, even if it meant him just watching me type gibberish into the command line, he didn’t mind.

Later on, at the age of 10, my father got me a copy of my first coding book called “Learn C++ in 12 Easy Steps”. Now, this was an absolute god damned lie. You cannot learn C++ in any configuration of 12 easy steps, 25 years on I still don’t think I fully know C++. And, while frustrating, this book really exploded what was mere curiosity about computers into a full-on passion.

There’s really a simple test for whether or not someone is truly going to fall in love with coding. It is the “Hello World” test. The first time you see those words printed out on screen as a result of a program that you’ve written you will know in your heart whether or not coding is for you. Because either you will be amazed and curious about how it is even possible for those letters to appear, or you will be disappointed that it took so much effort to get so little and such pointless output. I passed the test.

Within about 4 months of reading that book I had stopped going outside to play (kids still did that back then), setup my own BBS system (which was a sort of pre pre-internet version of the internet), and was starting to wrap my head around game programming. I had no idea that I could even get a job as a software engineer, no one told me it was a sensible career path and none of my friends even had computers. I just thought they were cool and wanted to learn what I could make them do.

Second, at this point in my life which was around the late 80’s to early 90’s, my mother was an up and coming Vice Principal in the Maryland / DC public school system. That should tell you everything you need to know about my mom. But in case you need me to unpack it for you: first, the DC public school system at this time was one of the roughest in the country. Second, Vice Principal is an enforcement role. They are the people who keep all the teachers and all the students in line. Third, my mother was extremely good at her job.

She brought drill sergeant discipline home and pushed me hard to excel with a combination of love and continual encouragement. I ended up with what is surely an unhealthy work ethic and extremely goal-oriented mindset. It has manifested in several ways that impact me to this day. For a long while it was common for me to start the work day at 5 AM and end around 11:30 PM. I don’t eat breakfast or lunch — don’t want to waste the hours when I could be working. I absolutely love competition, winning and taking on challenges, which results in me both playing way too many video games and reading articles about business rivals deep into the night. Finally, I make endless lists of things to do and have a continual and morbid feeling of unease and anxiety when I’m not completely busy.

So thanks ma for that!

Now, I’m definitely not complaining because she was giving me what I needed. You see, around this time in my life I started to get a bit more worldly — watching the news instead of cartoons, having more adult conversations. Because of my mom’s job, I also had one of the earliest internet connections and I could start to really understand what was going on in the world and my situation.

And I realized it was not good. And I mean personally, my prospects weren’t good. I was an 11-year-old black male, attending the 2nd to worst junior high school in my county. My parents were lower-middle class with no particularly promising social connections, little savings, and were deep in their own personal struggles at work and home. And society didn’t really seem to care. Depending on how you look at George Bush’s war on drugs, maybe it was even actively hostile to people like me.

It really all hit home for me one day when I was walking home from junior high school. First, I should note that my school had some rough edges. I remember things like the Vice Principal getting beaten up pretty badly by a student, classmates who got caught bringing guns to school, huge fights breaking out in the hallways. I don’t remember being too concerned by any of that. If you grow up with that kind of conflict it doesn’t phase you too much if it isn’t affecting you. Mind your own business.

But one day I’m walking home. Up ahead of me about 30 feet away, there is a group of about 8 kids from my school, and they are talking and joking when all of a sudden one asks the other “Hey so, last time you were at my house $15 turned up missing. You know anything about that $15?” Of course, the kid replies “no, I don’t know anything about that” and one thing led to another. Pretty soon 7 kids were on him, beating him, kicking him in the face, stomping him hard into the pavement. I’m not going to get any more graphic than that, but you can imagine what the outcome of a 7 on 1 fight looks like.

This wasn’t the first time I’d seen violence by any means, but it was right in front of me and it was intense. Later on, I would learn about Maslow’s hierarchy of needs, the five elements of motivation for humans starting with basic needs like physiological and safety, then moving on to the loftier goals of love, self-esteem, and self-actualization. Well, all that is just fancy words for the simple fact that nothing is more clarifying and motivating than hunger or the fear of your own safety.

When I got home it was the first time I really thought about my situation and the future. I thought about the world I saw online and in books and magazines. I thought about my interests, being so eclectic: basically an introvert-bookish-computer-sci-fi-video-game-band-anime nerd, and not fitting in at all. I imagined myself in this neighborhood for the next few years, attending the same high school as the kids in the mob I’d just seen beat someone half to death while laughing about it. And after surviving that, then working at a government job like my parents and the friends of my parents.

That just didn’t feel right. I realized that I did have hopes, things I wanted to learn about and be a part of in life. But every day they were slipping away, through no fault of my own, and I had to do something urgently.

Motivation can be about running toward something and it can also be about running from something. And that day, deep in my gut with all the disciplined energy that had been imbued in me by my hard-working mother, and all of the passion for learning granted to me by my father, I decided that I had to get the fuck out of that neighborhood no matter what it took.

Now, I don’t want to spend much longer talking about my pre-employment years so I promise to wrap this section up quickly. But I get asked a lot “well what happened next, how did you get out?” Long story short is that in the Maryland-DC area they have what are called magnet schools, and if you can pass a test sort of like the SAT, they will let you leave your current school zone and bus to a better one. Whether or not you think that’s a good or bad idea, or fair or unfair, it is what I had and it was a way out.

So, I studied for that test every day for about 3 months straight. I put off all other interests, school, computers, saxophone, friends. I just lived in study mode. In the end, I was placed 11th on the wait list, and very very luckily got in.

From that point on I commuted to high school about an hour every day to the nicer part of town. I should say that I was a bit culture-shocked when I turned out to be one of the few black kids in the high school program. Up until this point in my life I’d attended schools that were 90% black. White people were basically unicorns, now I’d gotten the situation totally flipped. But I’ll be honest, I just kept my head down and ignored this and kept working my ass off.

Through my mother’s former manager, I was very fortunate to discover a volunteer IT internship opening with the county school district. My boss then referred me to an open internship opportunity at NASA he’d heard about. A co-worker and MIT graduate I met in the NASA cafeteria then referred me to MIT’s new college prep program for minorities, called MITES. By the time I got to senior year I’d never taken a summer off, had built a respectable resume, scored well on the SAT (after 3 attempts), and was determined that I wanted to attend MIT. I applied and was accepted into other big-name schools, but I chose MIT specifically because it was known to be terrifyingly difficult. And it had become very much ingrained and proven to me that if I worked extremely hard my situation would just get better and better. So I always picked the most difficult challenges I could find.

MIT, as you all know, is located near Boston, Massachusetts. So I was again shocked to find out that there could be a place on Earth with even fewer black people than my high school program. People often ask me “what was it like” to make the transition. I’m not going to lie, Boston isn’t exactly a friendly place to begin with and being black there was not fun. I think I experienced the majority of racist incidents in my life in Boston: the usual name calling, driving while black, all that stuff. But MIT was so brutally difficult that I largely ignored all that so I could try to keep my head above water academically. There was no race in the trenches as far as I was concerned. Yes, getting called the N word every once in awhile stings but you don’t know true pain until you have to pass partial differential equations at MIT. Nothing has been as difficult for me since going to this school for undergrad. If you can survive partial differential equations at MIT you can survive anything.

So just to wrap all that up and put a bow on it. After graduation, the first thing I did was to buy sports car using a Microsoft offer letter as collateral. Then, I took a break, the first real time off work I’d had in a long time and headed back home for a few months. I hadn’t spent any significant length of time there in years and had lost touch with almost all my friends. So my hope was to go back and catch up with them.

Back home there was a restaurant called Roy Rogers. It’s kind of a cheap and run down fast food hamburger place but I remember it fondly because, at the time, they also sold Philly cheesesteaks. You could get them for about $3.50 which is about all I could afford when I was kid. At any rate, feeling nostalgic I drove my brand new Nissan 350z to the Roy Rogers, parked it out front, walked in, got a cheesesteak, sat down and started digging in. Roy Rogers is terrible but nothing beats the spice of nostalgia.

There was a janitor sweeping the floor when I sat down, his back was turned to me but he was slowly sweeping in my direction. And when he got to me he stopped sweeping, tapped me on the shoulder. As chance would have it, this was my absolute best friend from elementary school. For years, we did everything together: ride bikes, video games, basketball, saxophone. I hadn’t seen him since the 6th grade. He said, “so where have you been?” I told him I’d just graduated from MIT and gotten a job at Microsoft. He stared at me for about 3 seconds, then said, exactly these words in response “so I guess you are going to be rich then?” I said, “I guess. I hope so.” A moment passed. Then he said, “well shit,” turned around, and just kept sweeping away. That was pretty much it. I’ve never seen him again. Somehow he isn’t even on Facebook.

There are friends I try to keep in touch with from time to time. But as I learned: you can’t go home again. The vast majority of my old friends are still living in the same neighborhood, and it’s DC so they tend to have government jobs. They are postal workers, elementary and high school educators, mostly. Some are struggling actresses or models. A few are homeless, some got caught up in drugs or gangs, some are no longer with us. One guy works at Comcast. I feel the worst for him. As far as I know I’m the only person out in Silicon Valley from my neighborhood.

So when people ask me: what inspired you, what motivated you, this is all what runs through my head. It’s a simple middle-class story of hard work making your life better, perhaps made slightly more complex by an understanding that trade-offs and sacrifice are part of any significant achievement, and perhaps made even more complex by an understanding of how much luck plays a role.

Moving on, so that we can get to talking about some actual career and leadership moments: I had a semi-pro career in tech that lasted a few years during college but my first real professional job was at Microsoft. Nowadays people ask me “why on earth would you work at Microsoft?” Well there are two reasons.

The first is Bill Gates. Bill Gates was my hero. Bill Gates is still my hero even after I met him and got to see first-hand what a foul mouth he has (more on that later). I grew up using MS-DOS, Windows, and building my own PC hardware. I grew up during an era when Bill Gates said he wanted to revolutionize the world by putting a PC on every desk, he actually did, and became fantastically wealthy in the process. Computers were my lifeline to a better life and during that time Bill represented everything about how I imagined I’d achieve success. Even now Bill, via the Bill and Melinda Gates foundation, has gone on to attack global problems like malaria and HIV/AIDs and has used his influence to encourage other people of ridiculous wealth like Warren Buffet and Mark Zuckerberg to give back. He is also a huge Redditor.

The second is risk. People, particularly people from the Valley, ask me why I didn’t choose to work at Google or Facebook when I graduated. To answer that you’ve got to understand risk, and if the previous story about my upbringing wasn’t clear, I wanted some certainty that my hard work would lead to a better life. Google and Facebook were pre-IPO startups when I got out of college. Microsoft was a safe bet, and for a black kid coming out of P.G. County, wanting to buy a nice car, get his parents a house, this was a surefire win.

I know Diversity and Inclusiveness in the Valley are really red-hot topics now. One question is how can we attract more people of color. And this is a tough problem, but I’ll simply say that back in 2002 if you’d asked me to join a startup, even a Google or Facebook, I’d tell you you were insane. If you recited the common Silicon Valley mantra “fail fast” I’d tell you were insane. I felt for most of my early career that I didn’t have the luxury of failure, because any slip could land me right back where I started in P.G. County Maryland, maybe sweeping the floors at Roy Rogers. It wasn’t until I had a wide safety net of professional skills, a network, and frankly money, that I began to consider looking at startups. But a little of that fear of falling still exists in me, and I think exists in anyone who has had to climb up a few tough rungs to get where they are. Ironically, I’m now hiring heavily for Reddit and have lost diverse candidates to Fakebook for the exact same reason: it’s safer bet.

So anyway, I remember as the plane approached Seattle, looking out the window at the beautiful emerald green trees that surround the city, and thinking: I’m about to get paid.

References

Visible links
1. https://blog.devcolor.org/@nickcaldwell?source=post_header_lockup
2. http://www.reddit.com/r/place

HackerNewsBot debug: Calculated post rank: 106 - Loop: 374 - Rank min: 100 - Author rank: 100

Apple supplier shifts away from China

Key iPhone assembler Hon Hai, also known as Foxconn Technology Group, is increasing its presence in Vietnam and India
Article word count: 255

HN Discussion: https://news.ycombinator.com/item?id=19017786
Posted by ryansmccoy (karma: 158)
Post stats: Points: 142 - Comments: 45 - 2019-01-28T14:51:07Z

\#HackerNews #apple #away #china #from #shifts #supplier

---

Article content:




(Bloomberg) -- Key iPhone assembler Hon Hai Precision Industry Co., also known as Foxconn Technology Group, is increasing its presence in Vietnam and India amid the growing trade spat between the U.S. and China.

In a series of filings, the company said it had injected up to $213.5 million into an Indian unit between September and January for “long-term investment,” and is paying $16.5 million to Fuhua Co. Ltd. for the right to use 250,000 square meters of land in an industrial park in Vietnam’s northeastern Bac Giang Province for operations and sales. Hon Hai now owns 99.99% of the Indian unit, according to the filing.

While Hon Hai did not indicate the move was related to orders for Apple Inc., in a separate filing the Taiwanese manufacturer said it was selling on the Vietnamese land rights to a unit of Hong Kong-listed FIT Hon Teng, an iPhone cable and connector maker Foxconn controls. Hon Hai doesn’t make any iPhones in India, though its Hong Kong-listed subsidiary FIH Mobile assembles Xiaomi’s handsets in the country.

Some Taiwanese hardware makers have been shifting production out of China back home and to other countries, thanks to new tariffs on networking equipment and server motherboards. None have indicated Apple has asked them to move manufacturing elsewhere, even though U.S. President Donald Trump has threatened to eventually slap tariffs on all Chinese imports.

The manufacturers have repeatedly said they have existing operations in [1]overseas locations where they can further transfer production if needed.

A Foxconn spokesman did not immediately provide details on the moves.

References

Visible links
1. https://www.bloomberg.com/news/articles/2018-08-16/the-makers-of-iphones-and-laptops-are-prepping-for-a-trade-war

HackerNewsBot debug: Calculated post rank: 109 - Loop: 144 - Rank min: 100 - Author rank: 54

HTTP/3: from root to tip

Explore HTTP/3 from root to tip and discover the backstory of this new HTTP syntax that works on top of the IETF QUIC transport.
Article word count: 765

HN Discussion: https://news.ycombinator.com/item?id=19012705
Posted by jgrahamc (karma: 68624)
Post stats: Points: 170 - Comments: 44 - 2019-01-27T17:33:53Z

\#HackerNews #from #http #root #tip

---

Article content:




HTTP is the application protocol that powers the Web. It began life as the so-called HTTP/0.9 protocol in 1991, and by 1999 had evolved to HTTP/1.1, which was standardised within the IETF (Internet Engineering Task Force). HTTP/1.1 was good enough for a long time but the ever changing needs of the Web called for a better suited protocol, and HTTP/2 emerged in 2015. More recently it was announced that the IETF is intending to deliver a new version - HTTP/3. To some people this is a surprise and has caused a bit of confusion. If you donʼt track IETF work closely it might seem that HTTP/3 has come out of the blue. However, we can trace its origins through a lineage of experiments and evolution of Web protocols; specifically the QUIC transport protocol.

If youʼre not familiar with QUIC, my colleagues have done a great job of tackling different angles. Johnʼs [1]blog describes some of the real-world annoyances of todayʼs HTTP, Alessandroʼs [2]blog tackles the nitty-gritty transport layer details, and Nickʼs [3]blog covers how to get hands on with some testing. Weʼve collected these and more at [4]https://cloudflare-quic.com. And if that tickles your fancy, be sure to check out [5]quiche, our own open-source implementation of the QUIC protocol written in Rust.

HTTP/3 is the HTTP application mapping to the QUIC transport layer. This name was made official in the recent draft version 17 ([6]draft-ietf-quic-http-17), which was proposed in late October 2018, with discussion and rough consensus being formed during the IETF 103 meeting in Bangkok in November. HTTP/3 was previously known as HTTP over QUIC, which itself was previously known as HTTP/2 over QUIC. Before that we had HTTP/2 over gQUIC, and way back we had SPDY over gQUIC. The fact of the matter, however, is that HTTP/3 is just a new HTTP syntax that works on IETF QUIC, a UDP-based multiplexed and secure transport.

In this blog post weʼll explore the history behind some of HTTP/3ʼs previous names and present the motivation behind the most recent name change. Weʼll go back to the early days of HTTP and touch on all the good work that has happened along the way. If youʼre keen to get the full picture you can jump to the end of the article or open this [7]highly detailed SVG version.

An HTTP/3 layer cake

Setting the scene

Just before we focus on HTTP, it is worth reminding ourselves that there are two protocols that share the name QUIC. As we explained [8]previously, gQUIC is commonly used to identify Google QUIC (the original protocol), and QUIC is commonly used to represent the IETF standard-in-progress version that diverges from gQUIC.

Since its early days in the 90s, the web’s needs have changed. Weʼve had new versions of HTTP and added user security in the shape of Transport Layer Security (TLS). Weʼll only touch on TLS in this post, our other [9]blog posts are a great resource if you want to explore that area in more detail.

To help me explain the history of HTTP and TLS, I started to collate details of protocol specifications and dates. This information is usually presented in a textual form such as a list of bullets points stating document titles, ordered by date. However, there are branching standards, each overlapping in time and a simple list cannot express the real complexity of relationships. In HTTP, there has been parallel work that refactors core protocol definitions for easier consumption, extends the protocol for new uses, and redefines how the protocol exchanges data over the Internet for performance. When youʼre trying to join the dots over nearly 30 years of Internet history across different branching work streams you need a visualisation. So I made one - the Cloudflare Secure Web Timeline. (NB: Technically it is a [10]Cladogram, but the term timeline is more widely known).

I have applied some artistic license when creating this, choosing to focus on the successful branches in the IETF space. Some of the things not shown include efforts in the W3 Consortium [11]HTTP-NG working group, along with some exotic ideas that their authors are keen on explaining how to pronounce: [12]HMURR (pronounced ʼhammerʼ) and [13]WAKA (pronounced “wah-kah”).

In the next few sections Iʼll walk this timeline to explain critical chapters in the history of HTTP. To enjoy the takeaways from this post, it helps to have an appreciation of why standardisation is beneficial, and how the IETF approaches it. Therefore weʼll start with a very brief overview of that topic before returning to the timeline itself. Feel free to skip the next section if you are already familiar with the IETF.

Types of Internet standard

Generally, standards define common terms of reference, scope, constraint, applicability, and other considerations. Standards exist in many shapes and sizes, and can be informal (aka de facto) or formal (agreed/published by a Standards Defining Organisation such as IETF, ISO or MPEG). Standards are used in many fields, there is even a formal British Standard for making tea - BS 6008.

The early Web used HTTP and SSL protocol definitions that were published outside the IETF, these are marked as red lines on the Secure Web Timeline. The uptake of these protocols by clients and servers made them de facto standards.

At some point, it was decided to formalise these protocols (some motivating reasons are described in a later section). Internet standards are commonly defined in the IETF, which is guided by the informal principle of "rough consensus and running code". This is grounded in experience of developing and deploying things on the Internet. This is in contrast to a "clean room" approach of trying to develop perfect protocols in a vacuum.

IETF Internet standards are commonly known as RFCs. This is a complex area to explain so I recommend reading the blog post "[14]How to Read an RFC" by the QUIC Working Group Co-chair Mark Nottingham. A Working Group, or WG, is more or less just a mailing list.

Each year the IETF hold three meetings that provide the time and facilities for all WGs to meet in person if they wish. The agenda for these weeks can become very congested, with limited time available to discuss highly technical areas in depth. To overcome this, some WGs choose to also hold interim meetings in the months between the the general IETF meetings. This can help to maintain momentum on specification development. The QUIC WG has held several interim meetings since 2017, a full list is available on their [15]meeting page.

These IETF meetings also provide the opportunity for other IETF-related collections of people to meet, such as the [16]Internet Architecture Board or [17]Internet Research Task Force. In recent years, an [18]IETF Hackathon has been held during the weekend preceding the IETF meeting. This provides an opportunity for the community to develop running code and, importantly, to carry out interoperability testing in the same room with others. This helps to find issues in specifications that can be discussed in the following days.

For the purposes of this blog, the important thing to understand is that RFCs donʼt just spring into existence. Instead, they go through a process that usually starts with an IETF Internet Draft (I-D) format that is submitted for consideration of adoption. In the case where there is already a published specification, preparation of an I-D might just be a simple reformatting exercise. I-Ds have a 6 month active lifetime from the date of publish. To keep them active, new versions need to be published. In practice, there is not much consequence to letting an I-D elapse and it happens quite often. The documents continue to be hosted on the [19]IETF document’s website for anyone that wants to read them.

I-Ds are represented on the Secure Web Timeline as purple lines. Each one has a unique name that takes the form of draft-{author name}-{working group}-{topic}-{version}. The working group field is optional, it might predict IETF WG that will work on the piece and sometimes this changes. If an I-D is adopted by the IETF, or if the I-D was initiated directly within the IETF, the name is draft-ietf-{working group}-{topic}-{version}. I-Ds may branch, merge or die on the vine. The version starts at 00 and increases by 1 each time a new draft is released. For example, the 4th draft of an I-D will have the version 03. Any time that an I-D changes name, its version resets back to 00.

It is important to note that anyone can submit an I-D to the IETF; you should not consider these as standards. But, if the IETF standardisation process of an I-D does reach consensus, and the final document passes review, we finally get an RFC. The name changes again at this stage. Each RFC gets a unique number e.g. [20]RFC 7230. These are represented as blue lines on the Secure Web Timeline.

RFCs are immutable documents. This means that changes to the RFC require a completely new number. Changes might be done in order to incorporate fixes for errata (editorial or technical errors that were found and reported) or simply to refactor the specification to improve layout. RFCs may obsolete older versions (complete replacement), or just update them (substantively change).

All IETF documents are openly available on [21]http://tools.ietf.org. Personally I find the [22]IETF Datatracker a little more user friendly because it provides a visualisation of a documents progress from I-D to RFC.

Below is an example that shows the development of [23]RFC 1945 - HTTP/1.0 and it is a clear source of inspiration for the Secure Web Timeline.

IETF Datatracker view of RFC 1945

Interestingly, in the course of my work I found that the above visualisation is incorrect. It is missing [24]draft-ietf-http-v10-spec-05 for some reason. Since the I-D lifetime is 6 months, there appears to be a gap before it became an RFC, whereas in reality draft 05 was still active through until August 1996.

Exploring the Secure Web Timeline

With a small appreciation of how Internet standards documents come to fruition, we can start to walk the the Secure Web Timeline. In this section are a number of excerpt diagrams that show an important part of the timeline. Each dot represents the date that a document or capability was made available. For IETF documents, draft numbers are omitted for clarity. However, if you want to see all that detail please check out the [25]complete timeline.

HTTP began life as the so-called HTTP/0.9 protocol in 1991, and in 1994 the I-D [26]draft-fielding-http-spec-00 was published. This was adopted by the IETF soon after, causing the name change to [27]draft-ietf-http-v10-spec-00. The I-D went through 6 draft versions before being published as [28]RFC 1945 - HTTP/1.0 in 1996.

However, even before the HTTP/1.0 work completed, a separate activity started on HTTP/1.1. The I-D [29]draft-ietf-http-v11-spec-00 was published in November 1995 and was formally published as [30]RFC 2068 in 1997. The keen eyed will spot that the Secure Web Timeline doesnʼt quite capture that sequence of events, this is an unfortunate side effect of the tooling used to generate the visualisation. I tried to minimise such problems where possible.

An HTTP/1.1 revision exercise was started in mid-1997 in the form of [31]draft-ietf-http-v11-spec-rev-00. This completed in 1999 with the publication of [32]RFC 2616. Things went quiet in the IETF HTTP world until 2007. Weʼll come back to that shortly.

A History of SSL and TLS

Switching tracks to SSL. We see that the SSL 2.0 specification was released sometime around 1995, and that SSL 3.0 was released in November 1996. Interestingly, SSL 3.0 is described by [33]RFC 6101, which was released in August 2011. This sits in Historic category, which "is usually done to document ideas that were considered and discarded, or protocols that were already historic when it was decided to document them." according to the [34]IETF. In this case it is advantageous to have an IETF-owned document that describes SSL 3.0 because it can be used as a canonical reference elsewhere.

Of more interest to us is how SSL inspired the development of TLS, which began life as [35]draft-ietf-tls-protocol-00 in November 1996. This went through 6 draft versions and was published as [36]RFC 2246 - TLS 1.0 at the start of 1999.

Between 1995 and 1999, the SSL and TLS protocols were used to secure HTTP communications on the Internet. This worked just fine as a de facto standard. It wasnʼt until January 1998 that the formal standardisation process for HTTPS was started with the publication of I-D [37]draft-ietf-tls-https-00. That work concluded in May 2000 with the publication of [38]RFC 2616 - HTTP over TLS.

TLS continued to evolve between 2000 and 2007, with the standardisation of TLS 1.1 and 1.2. There was a gap of 7 years until work began on the next version of TLS, which was adopted as [39]draft-ietf-tls-tls13-00 in April 2014 and, after 28 drafts, completed as [40]RFC 8446 - TLS 1.3 in August 2018.

Internet standardisation process

After taking a small look at the timeline, I hope you can build a sense of how the IETF works. One generalisation for the way that Internet standards take shape is that researchers or engineers design experimental protocols that suit their specific use case. They experiment with protocols, in public or private, at various levels of scale. The data helps to identify improvements or issues. The work may be published to explain the experiment, to gather wider input or to help find other implementers. Take up of this early work by others may make it a de facto standard; eventually there may be sufficient momentum that formal standardisation becomes an option.

The status of a protocol can be an important consideration for organisations that may be thinking about implementing, deploying or in some way using it. A formal standardisation process can make a de facto standard more attractive because it tends to provide stability. The stewardship and guidance is provided by an organisation, such as the IETF, that reflects a wider range of experiences. However, it is worth highlighting that not all all formal standards succeed.

The process of creating a final standard is almost as important as the standard itself. Taking an initial idea and inviting contribution from people with wider knowledge, experience and use cases can to help produce something that will be of more use to a wider population. However, the standardisation process is not always easy. There are pitfalls and hurdles. Sometimes the process takes so long that the output is no longer relevant.

Each Standards Defining Organisation tends to have its own process that is geared around its field and participants. Explaining all of the details about how the IETF works is well beyond the scope of this blog. The IETFʼs "[41]How we work" page is an excellent starting point that covers many aspects. The best method to forming understanding, as usual, is to get involved yourself. This can be as easy as joining an email list or adding to discussion on a relevant GitHub repository.

Cloudflareʼs running code

Cloudflare is proud to be early an adopter of new and evolving protocols. We have a long record of adopting new standards early, such as [42]HTTP/2. We also test features that are experimental or yet to be final, like [43]TLS 1.3 and [44]SPDY.

In relation to the IETF standardisation process, deploying this running code on real networks across a diverse body of websites helps us understand how well the protocol will work in practice. We combine our existing expertise with experimental information to help improve the running code and, where it makes sense, feedback issues or improvements to the WG that is standardising a protocol.

Testing new things is not the only priority. Part of being an innovator is knowing when it is time to move forward and put older innovations in the rear view mirror. Sometimes this relates to security-oriented protocols, for example, Cloudflare [45]disabled SSLv3 by default due of the POODLE vulnerability. In other cases, protocols become superseded by a more technologically advanced one; Cloudflare [46]deprecated SPDY support in favour of HTTP/2.

The introduction and deprecation of relevant protocols are represented on the Secure Web Timeline as orange lines. Dotted vertical lines help correlate Cloudflare events to relevant IETF documents. For example, Cloudflare introduced TLS 1.3 support in September 2016, with the final document, [47]RFC 8446, being published almost two years later in August 2018.

Refactoring in HTTPbis

HTTP/1.1 is a very successful protocol and the timeline shows that there wasnʼt much activity in the IETF after 1999. However, the true reflection is that years of active use gave implementation experience that unearthed latent issues with [48]RFC 2616, which caused some interoperability issues. Furthermore, the protocol was extended by other RFCs like 2817 and 2818. It was decided in 2007 to kickstart a new activity to improve the HTTP protocol specification. This was called HTTPbis (where "bis" stems from Latin meaning "two", "twice" or "repeat") and it took the form of a new Working Group. The original [49]charter does a good job of describing the problems that were trying to be solved.

In short, HTTPbis decided to refactor [50]RFC 2616. It would incorporate errata fixes and buy in some aspects of other specifications that had been published in the meantime. It was decided to split the document up into parts. This resulted in 6 I-Ds published in December 2007:
\* draft-ietf-httpbis-p1-messaging
\* draft-ietf-httpbis-p2-semantics
\* draft-ietf-httpbis-p4-conditional
\* draft-ietf-httpbis-p5-range
\* draft-ietf-httpbis-p6-cache
\* draft-ietf-httpbis-p7-auth
The diagram shows how this work progressed through a lengthy drafting process of 7 years, with 27 draft versions being released, before final standardisation. In June 2014, the so-called RFC 723x series was released (where x ranges from 0 to 5). The Chair of the HTTPbis WG celebrated this achievement with the acclimation "[51]RFC2616 is Dead". If it wasnʼt clear, these new documents obsoleted the older [52]RFC 2616.

What does any of this have to do with HTTP/3?

While the IETF was busy working on the RFC 723x series the world didnʼt stop. People continued to enhance, extend and experiment with HTTP on the Internet. Among them were Google, who had started to experiment with something called SPDY (pronounced speedy). This protocol was touted as improving the performance of web browsing, a principle use case for HTTP. At the end of 2009 SPDY v1 was announced, and it was quickly followed by SPDY v2 in 2010.

I want to avoid going into the technical details of SPDY. Thatʼs a topic for another day. What is important, is to understand that SPDY took the core paradigms of HTTP and modified the interchange format slightly in order to gain improvements. With hindsight, we can see that HTTP has clearly delimited semantics and syntax. Semantics describe the concept of request and response exchanges including: methods, status codes, header fields (metadata) and bodies (payload). Syntax describe how to map semantics to bytes on the wire.

HTTP/0.9, 1.0 and 1.1 share many semantics. They also share syntax in the form of character strings that are sent over TCP connections. SPDY took HTTP/1.1 semantics and changed the syntax from strings to binary. This is a really interesting topic but we will go no further down that rabbit hole today.

Googleʼs experiments with SPDY showed that there was promise in changing HTTP syntax, and value in keeping the existing HTTP semantics. For example, keeping the format of URLs to use https:// avoided many problems that could have affected adoption.

Having seen some of the positive outcomes, the IETF decided it was time to consider what HTTP/2.0 might look like. The [53]slides from the HTTPbis session held during IETF 83 in March 2012 show the requirements, goals and measures of success that were set out. It is also clearly states that "HTTP/2.0 only signifies that the wire format isnʼt compatible with that of HTTP/1.x".

During that meeting the community was invited to share proposals. I-Ds that were submitted for consideration included [54]draft-mbelshe-httpbis-spdy-00, [55]draft-montenegro-httpbis-speed-mobility-00 and [56]draft-tarreau-httpbis-network-friendly-00. Ultimately, the SPDY draft was adopted and in November 2012 work began on [57]draft-ietf-httpbis-http2-00. After 18 drafts across a period of just over 2 years, [58]RFC 7540 - HTTP/2 was published in 2015. During this specification period, the precise syntax of HTTP/2 diverged just enough to make HTTP/2 and SPDY incompatible.

These years were a very busy period for the HTTP-related work at the IETF, with the HTTP/1.1 refactor and HTTP/2 standardisation taking place in parallel. This is in stark contrast to the many years of quiet in the early 2000s. Be sure to check out the full timeline to really appreciate the amount of work that took place.

Although HTTP/2 was in the process of being standardised, there was still benefit to be had from using and experimenting with SPDY. Cloudflare [59]introduced support for SPDY in August 2012 and only deprecated it in February 2018 when our statistics showed that less than 4% of Web clients continued to want SPDY. Meanwhile, we [60]introduced HTTP/2 support in December 2015, not long after the RFC was published, when our analysis indicated that a meaningful proportion of Web clients could take advantage of it.

Web client support of the SPDY and HTTP/2 protocols preferred the secure option of using TLS. The introduction of [61]Universal SSL in September 2014 helped ensure that all websites signed up to Cloudflare were able to take advantage of these new protocols as we introduced them.

gQUIC

Google continued to experiment between 2012 and 2015 they released SPDY v3 and v3.1. They also started working on gQUIC (pronounced, at the time, as quick) and the initial public specification was made available in early 2012.

The early versions of gQUIC made use of the SPDY v3 form of HTTP syntax. This choice made sense because HTTP/2 was not yet finished. The SPDY binary syntax was packaged into QUIC packets that could sent in UDP datagrams. This was a departure from the TCP transport that HTTP traditionally relied on. When stacked up all together this looked like:

SPDY over gQUIC layer cake

gQUIC used clever tricks to achieve performance. One of these was to break the clear layering between application and transport. What this meant in practice was that gQUIC only ever supported HTTP. So much so that gQUIC, termed "QUIC" at the time, was synonymous with being the next candidate version of HTTP. Despite the continued changes to QUIC over the last few years, which weʼll touch on momentarily, to this day, the term QUIC is understood by people to mean that initial HTTP-only variant. Unfortunately this is a regular source of confusion when discussing the protocol.

gQUIC continued to experiment and eventually switched over to a syntax much closer to HTTP/2. So close in fact that most people simply called it "HTTP/2 over QUIC". However, because of technical constraints there were some very subtle differences. One example relates to how the HTTP headers were serialized and exchanged. It is a minor difference but in effect means that HTTP/2 over gQUIC was incompatible with the IETFʼs HTTP/2.

Last but not least, we always need to consider the security aspects of Internet protocols. gQUIC opted not to use TLS to provide security. Instead Google developed a different approach called QUIC Crypto. One of the interesting aspects of this was a new method for speeding up security handshakes. A client that had previously established a secure session with a server could reuse information to do a "zero round-trip time", or 0-RTT, handshake. 0-RTT was later incorporated into TLS 1.3.

Are we at the point where you can tell me what HTTP/3 is yet?

Almost.

By now you should be familiar with how standardisation works and gQUIC is not much different. There was sufficient interest that the Google specifications were written up in I-D format. In June 2015 [62]draft-tsvwg-quic-protocol-00, entitled "QUIC: A UDP-based Secure and Reliable Transport for HTTP/2" was submitted. Keep in mind my earlier statement that the syntax was almost-HTTP/2.

Google [63]announced that a Bar BoF would be held at IETF 93 in Prague. For those curious about what a "Bar BoF" is, please consult [64]RFC 6771. Hint: BoF stands for Birds of a Feather.

The outcome of this engagement with the IETF was, in a nutshell, that QUIC seemed to offer many advantages at the transport layer and that it should be decoupled from HTTP. The clear separation between layers should be re-introduced. Furthermore, there was a preference for returning back to a TLS-based handshake (which wasnʼt so bad since TLS 1.3 was underway at this stage, and it was incorporating 0-RTT handshakes).

About a year later, in 2016, a new set of I-Ds were submitted:

Hereʼs where another source of confusion about HTTP and QUIC enters the fray. [65]draft-shade-quic-http2-mapping-00 is entitled "HTTP/2 Semantics Using The QUIC Transport Protocol" and it describes itself as "a mapping of HTTP/2 semantics over QUIC". However, this is a misnomer. HTTP/2 was about changing syntax while maintaining semantics. Furthermore, "HTTP/2 over gQUIC" was never an accurate description of the syntax either, for the reasons I outlined earlier. Hold that thought.

This IETF version of QUIC was to be an entirely new transport protocol. Thatʼs a large undertaking and before diving head-first into such commitments, the IETF likes to gauge actual interest from its members. To do this, a formal [66]Birds of a Feather meeting was held at the IETF 96 meeting in Berlin in 2016. I was lucky enough to attend the session in person and the [67]slides donʼt give it justice. The meeting was attended by hundreds, as shown by Adam Roachʼs [68]photograph. At the end of the session consensus was reached; QUIC would be adopted and standardised at the IETF.

The first IETF QUIC I-D for mapping HTTP to QUIC, [69]draft-ietf-quic-http-00, took the Ronseal approach and simplified its name to "HTTP over QUIC". Unfortunately, it didnʼt finish the job completely and there were many instances of the term HTTP/2 throughout the body. Mike Bishop, the I-Ds new editor, identified this and started to fix the HTTP/2 misnomer. In the 01 draft, the description changed to "a mapping of HTTP semantics over QUIC".

Gradually, over time and versions, the use of the term "HTTP/2" decreased and the instances became mere references to parts of [70]RFC 7540. Roll forward two years to October 2018 and the I-D is now at version 16. While HTTP over QUIC bares similarity to HTTP/2 it ultimately is an independent, non-backwards compatible HTTP syntax. However, to those that donʼt track IETF development very closely (a very very large percentage of the Earthʼs population), the document name doesnʼt capture this difference. One of the main points of standardisation is to aid communication and interoperability. Yet a simple thing like naming is a major contributor to confusion in the community.

Recall what was said in 2012, "HTTP/2.0 only signifies that the wire format isnʼt compatible with that of HTTP/1.x". The IETF followed that existing cue. After much deliberation in the lead up to, and during, IETF 103, consensus was reached to rename "HTTP over QUIC" to HTTP/3. The world is now in a better place and we can move on to more important debates.

But RFC 7230 and 7231 disagree with your definition of semantics and syntax!

Sometimes document titles can be confusing. The present HTTP documents that describe syntax and semantics are:
\* [71]RFC 7230 - Hypertext Transfer Protocol (HTTP/1.1): Message Syntax and Routing
\* [72]RFC 7231 - Hypertext Transfer Protocol (HTTP/1.1): Semantics and Content
It is possible to read too much into these names and believe that fundamental HTTP semantics are specific for versions of HTTP i.e. HTTP/1.1. However, this is an unintended side effect of the HTTP family tree. The good news is that the HTTPbis Working Group are trying to address this. Some brave members are going through another round of document revision, as Roy Fielding put it, "one more time!". This work is underway right now and is known as the HTTP Core activity (you may also have heard of this under the moniker HTTPtre or HTTPter; naming things is hard). This will condense the six drafts down to three:
\* HTTP Semantics (draft-ietf-httpbis-semantics)
\* HTTP Caching (draft-ietf-httpbis-caching)
\* HTTP/1.1 Message Syntax and Routing (draft-ietf-httpbis-messaging)
Under this new structure, it becomes more evident that HTTP/2 and HTTP/3 are syntax definitions for the common HTTP semantics. This doesnʼt mean they donʼt have their own features beyond syntax but it should help frame discussion going forward.

Pulling it all together

This blog post has taken a shallow look at the standardisation process for HTTP in the IETF across the last three decades. Without touching on many technical details, Iʼve tried to explain how we have ended up with HTTP/3 today. If you skipped the good bits in the middle and are looking for a one liner here it is: HTTP/3 is just a new HTTP syntax that works on IETF QUIC, a UDP-based multiplexed and secure transport. There are many interesting technical areas to explore further but that will have to wait for another day.

In the course of this post, we explored important chapters in the development of HTTP and TLS but did so in isolation. We close out the blog by pulling them all together into the complete Secure Web Timeline presented below. You can use this to investigate the detailed history at your own comfort. And for the super sleuths, be sure to check out the [73]full version including draft numbers.

[74]Learn more

[75]comments powered by

Disqus

References

Visible links
1. https://blog.cloudflare.com/the-quicening/
2. https://blog.cloudflare.com/the-road-to-quic/
3. https://blog.cloudflare.com/head-start-with-quic/
4. https://cloudflare-quic.com/
5. https://blog.cloudflare.com/enjoy-a-slice-of-quic-and-rust/
6. https://tools.ietf.org/html/draft-ietf-quic-http-17
7. https://blog.cloudflare.com/content/images/2019/01/web_timeline_large1.svg
8. https://blog.cloudflare.com/the-road-to-quic/
9. https://blog.cloudflare.com/tag/tls/
10. https://en.wikipedia.org/wiki/Cladogram
11. https://www.w3.org/Protocols/HTTP-NG/
12. https://blog.jgc.org/2012/12/speeding-up-http-with-minimal-protocol.html
13. https://github.com/HTTPWorkshop/workshop2017/blob/master/talks/waka.pdf
14. https://www.ietf.org/blog/how-read-rfc/
15. https://datatracker.ietf.org/wg/quic/meetings/
16. https://www.iab.org/
17. https://irtf.org/
18. https://www.ietf.org/how/runningcode/hackathons/
19. https://datatracker.ietf.org/doc/recent
20. https://tools.ietf.org/html/rfc7230
21. http://tools.ietf.org/
22. https://datatracker.ietf.org/
23. https://tools.ietf.org/html/rfc1945
24. https://tools.ietf.org/html/draft-ietf-http-v10-spec-05
25. https://blog.cloudflare.com/content/images/2019/01/web_timeline_large1.svg
26. https://tools.ietf.org/html/draft-fielding-http-spec-00
27. https://tools.ietf.org/html/draft-ietf-http-v10-spec-00
28. https://tools.ietf.org/html/rfc1945
29. https://tools.ietf.org/html/draft-ietf-http-v11-spec-00
30. https://tools.ietf.org/html/rfc2068
31. https://tools.ietf.org/html/draft-ietf-http-v11-spec-rev-00
32. https://tools.ietf.org/html/rfc2616
33. https://tools.ietf.org/html/rfc6101
34. https://www.ietf.org/blog/iesg-statement-designating-rfcs-historic/?primary_topic=7&
35. https://tools.ietf.org/html/draft-ietf-tls-protocol-00
36. https://tools.ietf.org/html/rfc2246
37. https://tools.ietf.org/html/draft-ietf-tls-https-00
38. https://tools.ietf.org/html/rfc2616
39. https://tools.ietf.org/html/draft-ietf-tls-tls13-00
40. https://tools.ietf.org/html/rfc8446
41. https://www.ietf.org/how/
42. https://blog.cloudflare.com/introducing-http2/
43. https://blog.cloudflare.com/introducing-tls-1-3/
44. https://blog.cloudflare.com/introducing-spdy/
45. https://blog.cloudflare.com/sslv3-support-disabled-by-default-due-to-vulnerability/
46. https://blog.cloudflare.com/deprecating-spdy/
47. https://tools.ietf.org/html/rfc8446
48. https://tools.ietf.org/html/rfc2616
49. https://tools.ietf.org/wg/httpbis/charters?item=charter-httpbis-2007-10-23.txt
50. https://tools.ietf.org/html/rfc2616
51. https://www.mnot.net/blog/2014/06/07/rfc2616_is_dead
52. https://tools.ietf.org/html/rfc2616
53. https://github.com/httpwg/wg-materials/blob/gh-pages/ietf83/HTTP2.pdf
54. https://tools.ietf.org/html/draft-mbelshe-httpbis-spdy-00
55. https://tools.ietf.org/html/draft-montenegro-httpbis-speed-mobility-00
56. https://tools.ietf.org/html/draft-tarreau-httpbis-network-friendly-00
57. https://tools.ietf.org/html/draft-ietf-httpbis-http2-00
58. https://tools.ietf.org/html/rfc7540
59. https://blog.cloudflare.com/spdy-now-one-click-simple-for-any-website/
60. https://blog.cloudflare.com/introducing-http2/
61. https://blog.cloudflare.com/introducing-universal-ssl/
62. https://tools.ietf.org/html/draft-tsvwg-quic-protocol-00
63. https://groups.google.com/a/chromium.org/forum/#!topic/proto-quic/otGKB4ytAyc
64. https://tools.ietf.org/html/rfc6771
65. https://tools.ietf.org/html/draft-shade-quic-http2-mapping-00
66. https://www.ietf.org/how/bofs/
67. https://datatracker.ietf.org/meeting/96/materials/slides-96-quic-0
68. https://www.flickr.com/photos/adam-roach/28343796722/in/photostream/
69. https://tools.ietf.org/html/draft-ietf-quic-http-00
70. https://tools.ietf.org/html/rfc7540
71. https://tools.ietf.org/html/rfc7230
72. https://tools.ietf.org/html/rfc7231
73. https://blog.cloudflare.com/content/images/2019/01/web_timeline_large1.svg
74. https://www.cloudflare.com/
75. http://disqus.com/

HackerNewsBot debug: Calculated post rank: 128 - Loop: 215 - Rank min: 100 - Author rank: 98

It’s Time to Move on from Two Phase Commit

The two-phase commit protocol (2PC) has been used in enterprise software systems for over three decades . It has been an an incredibly impa...
Article word count: 3534

HN Discussion: https://news.ycombinator.com/item?id=18999520
Posted by evanweaver (karma: 986)
Post stats: Points: 260 - Comments: 112 - 2019-01-25T16:31:09Z

\#HackerNews #commit #from #its #move #phase #time #two

---

Article content:




The two-phase commit protocol (2PC) has been [1]used in enterprise software systems for over three decades. It has been an an incredibly impactful protocol for ensuring atomicity and durability of transactions that access data in multiple partitions or shards. It is used everywhere --- both in older “venerable” distributed systems, database systems, and file systems such as Oracle, IBM DB2, PostgreSQL, and Microsoft TxF (transactional NTFS), and in younger “millennial” systems such as MariaDB, TokuDB, VoltDB, Cloud Spanner, Apache Flink, Apache Kafka, and Azure SQL Database. If your system supports ACID transactions across shards/partitions/databases, there’s a high probability that it is running 2PC (or some variant thereof) under the covers. [Sometimes it’s even “over the covers” --- older versions of MongoDB [2]required users to implement 2PC for multi-document transactions in application code.]

In this post, we will first describe 2PC: how it works and what problems it solves. Then, we will show some major issues with 2PC and how modern systems attempt to get around these issues. Unfortunately, these attempted solutions cause other problems to emerge. In the end, I will make the case that the next generation of distributed systems should avoid 2PC, and how this is possible.

There are many variants of 2PC, but the basic protocol works as follows:

Background assumption:The work entailed by a transaction has already been divided across all of the shards/partitions that store data accessed by that transaction. We will refer to the effort performed at each shard as being performed by the “worker” for that shard. Each worker is able to start working on its responsibilities for a given transaction independently of each other. The 2PC protocol begins at the end of transaction processing, when the transaction is ready to “commit”. It is initiated by a single, coordinator machine (which may be one of the workers involved in that transaction).

The basic flow of the 2PC protocol is shown in the figure below. [The protocol begins at the top of the figure and then proceeds in a downward direction.]

Phase 1: A coordinator asks each worker whether they have successfully completed their responsibilities for that transaction and are ready to commit. Each worker responds ‘yes’ or ‘no’.

Phase 2: The coordinator counts all the responses. If every worker responded ‘yes’, then the transaction will commit. Otherwise, it will abort. The coordinator sends a message to each worker with the final commit decision and receives an acknowledgement back.

This mechanism ensures the atomicity property of transactions: either the entire transaction will be reflected in the final state of the system, or none of it. If even just a single worker cannot commit, then the entire transaction will be aborted. In other words: each worker has “veto-power” for a transaction.

It also ensures transaction durability. Each worker ensures that all of the writes of a transaction have been durably written to storage prior to responding ‘yes’ in phase 1. This gives the coordinator freedom to make a final decision about a transaction without concern for the fact that a worker may fail after voting ‘yes’. [In this post, we are being purposefully vague when using the term “durable writes” --- this term can either refer to writing to local non-volatile storage or, alternatively, replicating the writes to enough locations for it to be considered “durable”.]

In addition to durably writing the writes that are directly required by the transaction, the protocol itself requires additional writes that must be made durable before it can proceed. For example, a worker has veto power until the point it votes ‘yes’ in phase 1. After that point, it cannot change its vote. But what if it crashes right after voting ‘yes’? When it recovers it might not know that it voted ‘yes’, and still think it has veto power and go ahead and abort the transaction. To prevent this, it must write its vote durably before sending the ‘yes’ vote back to the coordinator. [In addition to this example, in standard 2PC, there are two other writes that are made durable prior to sending messages that are part of the protocol.]

There are two major problems with 2PC. The first is well known, and discussed in every reputable textbook that presents 2PC. The second is much less well known, but a major problem nonetheless.

The well-known problem is referred to as the “blocking problem”. This happens when every worker has voted ‘yes’, but the coordinator fails before sending a message with the final decision to at least one worker. The reason why this is a problem is that by voting ‘yes’, each worker has removed its power to veto the transaction. However, the coordinator still has absolute power to decide the final state of a transaction. If the coordinator fails before sending a message with the final decision to at least one worker, the workers cannot get together to make a decision amongst themselves --- they can’t abort because maybe the coordinator decided to commit before it failed, and they can’t commit because maybe the coordinator decided to abort before it failed. Thus, they have to block --- wait until the coordinator recovers --- in order to find out the final decision. In the meantime, they cannot process transactions that conflict with the stalled transaction since the final outcome of the writes of that transaction are yet to be determined.

There are two categories of work-arounds to the blocking problem. The first category of work-around modifies the core protocol in order to eliminate the blocking problem. Unfortunately, these modifications reduce the performance --- typically by adding an extra round of communication --- and thus are rarely used in practice. The second category keeps the protocol in tact but reduces the probability of the types of coordinator failure than can lead to the blocking program --- for example, by running 2PC over replica consensus protocols and ensuring that important state for the protocol is replicated at all times. Unfortunately, once again, these work-arounds reduce performance, since the protocol requires that these replica consensus rounds occur sequentially, and thus they may add significant latency to the protocol.

The lesser-known problem is what I call the “cloggage problem”. 2PC occurs after transaction is processed, and thus necessarily increases the latency of the transaction by an amount equal to the time it takes to run the protocol. This latency increase alone can already be an issue for many applications, but a potentially larger issue is that worker nodes do not know the final outcome of a transaction until mid-way through the second phase. Until they know the final outcome, they have to be prepared for the possibility that it might abort, and thus they typically prevent conflicting transactions from making progress until they are certain that the transaction will commit. These blocked transactions in turn block other transactions from running, and so on, until 2PC completes and all of the blocked transactions can resume. This cloggage further increases the average transaction latency and also decreases transactional throughput.

To summarize the problems we discussed above: 2PC poisons a system along four dimensions: latency (the time of the protocol plus the stall time of conflicting transactions), throughput (because it prevents conflicting transactions from running during the protocol), scalability (the larger the system, the more likely transactions become multi-partition and have to pay the throughput and latency costs of 2PC), and availability (the blocking problem we discussed above). Nobody likes 2PC, but for decades, people have assumed that it is a necessary evil.

For over three decades, we’ve been stuck with two-phase commit in sharded systems. People are aware of the performance, scalability, and availability problems it introduces, but nonetheless continue on, with no obvious better alternative.

The truth is, if we would just architect our systems differently, the need for 2PC would vanish. There have been some attempts to accomplish this --- both in academia (such as [3]this SIGMOD 2016 paper) and industry. However, these attempts typically work by avoiding multi-sharded transactions in the first place, such as by repartitioning data in advance of a transaction so that it is no longer multi-sharded. Unfortunately, this repartitioning reduces performance of the system in other ways.

What I am calling for is a deeper type of change in the way we architect distributed systems. I insist that systems should still be able to process multi-sharded transactions --- with all the ACID guarantees and what that entails --- such as atomicity and durability --- but with much simpler and faster commit protocols.

It all comes down to a fundamental assumption that has been present in our systems for decades: a transaction may abort at any time and for any reason. Even if I run the same transaction on the same initial system state … if I run it at 2:00PM it may commit, but at 3:00 it may abort.

The are several reasons why most architects believe we need this assumption. First, a machine may fail at anytime --- including in the middle of a transaction. Upon recovery, it is generally impossible to recreate all of the state of that transaction that was in volatile memory prior to the failure. As a result, it is seemingly impossible to pick up where the transaction left off prior to the failure. Therefore, the system aborts all transactions that were in progress at the time of the failure. Since a failure can occur at any time, this means that a transaction may abort at any time.

Second, most concurrency control protocols require the ability to abort a transaction at any time. Optimistic protocols perform a “validation” phase after processing a transaction. If validation fails, the transaction aborts. Pessimistic protocols typically use locks to prevent concurrency anomalies. This use of locks may lead to deadlock, which is resolved by aborting (at least) one of the deadlocked transactions. Since deadlock may be discovered at any time, the transaction needs to retain the ability to abort at any time.

If you look carefully at the two-phase commit protocol, you will see that this arbitrary potential to abort a transaction is the primary source of complexity and latency in the protocol. Workers cannot easily tell each other whether they will commit or not, because they might still fail after this point (before the transaction is committed) and want to abort this transaction during recovery. Therefore, they have to wait until the end of transaction processing (when all important state is made durable) and proceed in the necessary two phases: in the first phase, each worker publically relinquishes its control to abort a transaction, and only then can the second phase occur in which a final decision is made and disseminated.

In my opinion we need to remove veto power from workers and architect systems in which the system does not have freedom to abort a transaction whenever it wants during its execution. Only logic within a transaction should be allowed to cause a transaction to abort. If it is theoretically possible to commit a transaction given an current state of the database, that transaction must commit, no matter what types of failures occur. Furthermore, there must not be race conditions relative to other concurrently running transactions that can affect the final commit/abort state of a transaction.

Removing abort flexibility sounds hard. We’ll discuss soon how to accomplish this. But first let’s observe how the commit protocol changes if transactions don’t have abort flexibility.

Let’s look at two examples:

In the first example, assume that the worker for the shard that stores the value for variable X is assigned a single task for a transaction: change the value of X to 42. Assume (for now) that there are no integrity constraints or triggers defined on X (which may prevent the system from setting X to 42). In such a case, that worker is never given the power to be able to abort the transaction. No matter what happens, that worker must change X to 42. If that worker fails, it must change X to 42 after it recovers. Since it never has any power to abort, there is no need to check with that worker during the commit protocol to see if it will commit.

In the second example, assume that the worker for the shard that stores the value for variables Y and Z is assigned two tasks for a transaction: subtract 1 from the previous value of Y and set Z to the new value of Y. Furthermore, assume that there is an integrity constraint on Y that states that Y can never go below 0 (e.g., if it represents the inventory of an item in a retail application). Therefore, this worker has to run the equivalent of the following code:

IF (Y > 0)

Subtract 1 from Y

ELSE

ABORT the transaction

Z = Y

This worker must be given the power to abort the transaction since this required by the logic of the application. However, this power is limited. Only if the initial value of Y was 0 can this worker abort the transaction. Otherwise, it has no choice but to commit. Therefore, it doesn’t have to wait until it has completed the transaction code before knowing whether it will commit or not. On the contrary: as soon as it has finished executing the first line of code in the transaction, it already knows its final commit/abort decision. This implies that the commit protocol will be able to start much earlier relative to 2PC.

Let’s now combine these two examples into a single example in which a transaction is being performed by two workers --- one of them is doing the work described in the first example, and the other one doing the work described in the second example. Since we are guaranteeing atomicity, the first worker cannot simply blindly set X to 42. Rather, it’s own work must also be dependent on the value of Y. In effect, it’s transaction code becomes:

temp = Do_Remote_Read(Y)

if (temp > 0)

X = 42

Note that if the first worker’s code is written in this way, the code for the other worker can be simplified to just:

IF (Y > 0)

Subtract 1 from Y

Z = Y

By writing the transaction code in this way, we have removed explicit abort logic from both workers. Instead, both workers have if statements that check for the constraint that would have caused the original transaction to abort. If the original transaction would have aborted, both workers end up doing nothing. Otherwise, both workers change the values of their local state as required by the transaction logic.

The important thing to note at this point is that the need for a commit protocol has been totally eliminated in the above code. The system is not allowed to abort a transaction for any reason other than conditional logic defined by application code on a given state of the data. And all workers condition their writes on this same conditional logic so that they can all independently decide to “do nothing” in those situations where a transaction cannot complete as a result of current system state. Thus, all possibility of a transaction abort has been removed, and there is no need for any kind of distributed protocol at the end of transaction processing to make a combined final decision about the transaction. All of the problems of 2PC have been eliminated. There is no blocking problem because there is no coordinator. And there is no cloggage problem, because all necessary checks are overlapped with transaction processing instead of after it completes.

Moreover, as long as the system is not allowed to abort a transaction for any reason other than the conditional application logic based on input data state, it is always possible to rewrite any transaction as we did above in order to replace abort logic in the code with if statements that conditionally check the abort conditions. Furthermore, it is possible to accomplish this without actually rewriting application code. [The details of how to do this are out of scope for this post, but to summarize at a high level: shards can set special system-owned boolean flags when they have completed any conditional logic that could cause an abort, and it is these boolean flags that are remotely read from other shards.]

In essence: there are two types of aborts that are possible in transaction processing systems: (1) Those that are caused by the state of the data and (2) Those that are caused by the system itself (e.g. failures or deadlocks). Category (1) can always be written in terms of conditional logic on the data as we did above. So if you can eliminate category (2) aborts, the commit protocol can be eliminated.

So now, all we have to do is explain how to eliminate category (2) aborts.

I have spent almost an entire decade designing systems that do not allow system-induced aborts. Examples of such systems are [4]Calvin, [5]CalvinFS, [6]Orthrus, [7]PVW, and a [8]system that processes transactions lazily. The impetus for this feature came from the first of these projects --- Calvin --- because of its status of being a deterministic database system. A deterministic database guarantees that there is only one possible final state of the data in the database given a defined set of input requests. It is therefore possible to send the same input to two distinct replicas of the system and be certain that the replicas will process this input independently and end up in the same final state, without any possibility of divergence.

System-induced aborts such as system failure or concurrency control race conditions are fundamentally nondeterministic events. It is very possible that one replica will fail or enter a race condition while the other replica will not. If these nondeterministic events were allowed to result in an a transaction to abort, then one replica may abort a transaction while the other one would commit --- a fundamental violation of the deterministic guarantee. Therefore, we had to design Calvin in a way that failures and race conditions cannot result in a transaction to abort. For concurrency control, Calvin used pessimistic locking with a deadlock avoidance technique that ensured that the system would never get into a situation where it had to abort a transaction due to deadlock. In the face of a system failure, Calvin did not pick up a transaction exactly where it left off (because of the loss of volatile memory during the failure). Nonetheless, it was able to bring the processing of that transaction to completion without having to abort it. It accomplished this via restarting the transaction from the same original input.

Neither of these solutions --- neither deadlock avoidance nor transaction restart upon a failure --- are limited to being used in deterministic database systems. [Transaction restart gets a little tricky in nondeterministic systems if some of the volatile state associated with a transaction that was lost during a failure was observed by other machines that did not fail. But there are simple ways to solve this problem that are out of scope for this post.] Indeed, some of the other systems I linked to above are nondeterministic systems. Once we realized the power that comes with removing system-level aborts, we built this feature into every system we built after the Calvin project --- even the nondeterministic systems.

I see very little benefit in system architects making continued use of 2PC in sharded systems moving forward. I believe that removing system-induced aborts and rewriting state-induced aborts is the better way forward. Deterministic database systems such as Calvin or [9]FaunaDB always remove system-induced aborts anyway, and thus usually can avoid 2PC as we described above. But it is a huge waste to limit this benefit to only deterministic databases. It is not hard to remove system-induced aborts from nondeterministic systems. Recent projects have shown that it is even possible to remove system-induced aborts in systems that use concurrency control techniques other than pessimistic concurrency control. For example, both the PVW and the lazy transaction processing systems we linked to above use a variant of multi-versioned concurrency control. And FaunaDB uses a variant of optimistic concurrency control.

In my opinion there is very little excuse to continue with antiquated assumptions regarding the need for system-induced aborts in the system. In the old days when systems ran on single machines, such assumptions were justifiable. However, in modern times, where many systems need to scale to multiple machines that can fail independently of each other, these assumptions require expensive coordination and commit protocols such as 2PC. The performance problems of 2PC has been a major force behind the rise of non-ACID compliant systems that give up important guarantees in order to achieve better scalability, availability, and performance. 2PC is just too slow --- it increases the latency of all transactions --- not just by the length of the protocol itself, but also by preventing transactions that access the same set of data from running concurrently. 2PC also limits scalability (by reducing concurrency) and availability (the blocking problem we discussed above). The way forward is clear: we need to reconsider antiquated assumptions when designing our systems and say “good-bye” to two phase commit!

References

Visible links
1. https://dl.acm.org/citation.cfm?id=7266
2. https://docs.mongodb.com/v3.4/tutorial/perform-two-phase-commits/
3. https://cs.uwaterloo.ca/~kdaudjee/courses/cs848/papers/non2PC.pdf
4. http://www.cs.umd.edu/~abadi/papers/calvin-sigmod12.pdf
5. http://www.cs.umd.edu/~abadi/papers/calvinfs.pdf
6. http://www.cs.umd.edu/~abadi/papers/orthrus-sigmod16.pdf
7. http://www.cs.umd.edu/~abadi/papers/early-write-visibility.pdf
8. http://www.cs.umd.edu/~abadi/papers/lazy-xacts.pdf
9. https://fauna.com/

HackerNewsBot debug: Calculated post rank: 210 - Loop: 416 - Rank min: 100 - Author rank: 57