Journalist/developer. Interactive editor @frontlinepbs. Builder of @HomicideWatch. Sinophile for fun. Past: @WBUR, @NPR, @NewsHour.
1683 stories

Election Screen Time

1 Comment and 4 Shares
Feels like I picked a bad year to try to start having a healthy relationship with political news.
Read the whole story
7 days ago
Boston, MA
Share this story
1 public comment
8 days ago
Feels like I picked a bad year to try to start having a healthy relationship with political news.

Git scraping: track changes over time by scraping to a Git repository

1 Share

Git scraping is the name I've given a scraping technique that I've been experimenting with for a few years now. It's really effective, and more people should use it.

The internet is full of interesting data that changes over time. These changes can sometimes be more interesting than the underlying static data - The @nyt_diff Twitter account tracks changes made to New York Times headlines for example, which offers a fascinating insight into that publication's editorial process.

We already have a great tool for efficiently tracking changes to text over time: Git. And GitHub Actions (and other CI systems) make it easy to create a scraper that runs every few minutes, records the current state of a resource and records changes to that resource over time in the commit history.

Here's a recent example. Fires continue to rage in California, and the CAL FIRE website offers an incident map showing the latest fire activity around the state.

Firing up the Firefox Network pane, filtering to requests triggered by XHR and sorting by size, largest first reveals this endpoint:

That's a 241KB JSON endpoints with full details of the various fires around the state.

So... I started running a git scraper against it. My scraper lives in the simonw/ca-fires-history repository on GitHub.

Every 20 minutes it grabs the latest copy of that JSON endpoint, pretty-prints it (for diff readability) using jq and commits it back to the repo if it has changed.

This means I now have a commit log of changes to that information about fires in California. Here's an example commit showing that last night the Zogg Fires percentage contained increased from 90% to 92%, the number of personnel involved dropped from 968 to 798 and the number of engines responding dropped from 82 to 59.

Screenshot of a diff against the Zogg Fires, showing personnel involved dropping from 968 to 798, engines dropping 82 to 59, water tenders dropping 31 to 27 and percent contained increasing from 90 to 92.

The implementation of the scraper is entirely contained in a single GitHub Actions workflow. It's in a file called .github/workflows/scrape.yml which looks like this:

name: Scrape latest data

    - cron:  '6,26,46 * * * *'

    runs-on: ubuntu-latest
    - name: Check out this repo
      uses: actions/checkout@v2
    - name: Fetch latest data
      run: |-
        curl | jq . > incidents.json
    - name: Commit and push if it changed
      run: |-
        git config "Automated"
        git config ""
        git add -A
        timestamp=$(date -u)
        git commit -m "Latest data: ${timestamp}" || exit 0
        git push

That's not a lot of code!

It runs on a schedule at 6, 26 and 46 minutes past the hour - I like to offset my cron times like this since I assume that the majority of crons run exactly on the hour, so running not-on-the-hour feels polite.

The scraper itself works by fetching the JSON using curl, piping it through jq . to pretty-print it and saving the result to incidents.json.

The "commit and push if it changed" block uses a pattern that commits and pushes only if the file has changed. I wrote about this pattern in this TIL a few months ago.

I have a whole bunch of repositories running git scrapers now. I've been labeling them with the git-scraping topic so they show up in one place on GitHub (other people have started using that topic as well).

I've written about some of these in the past:

I hope that by giving this technique a name I can encourage more people to add it to their toolbox. It's an extremely effective way of turning all sorts of interesting data sources into a changelog over time.

Comment thread on this post over on Hacker News.

Read the whole story
11 days ago
Boston, MA
Share this story

Clear Language on Slavery


I’ve posted before about how the language we’ve been conditioned to use about slavery and the Civil War obscures reality. From historian Michael Todd Landis:

Likewise, scholar Edward Baptist (Cornell) has provided new terms with which to speak about slavery. In his 2014 book The Half Has Never Been Told: Slavery and the Making of American Capitalism (Basic Books), he rejects “plantations” (a term pregnant with false memory and romantic myths) in favor of “labor camps”; instead of “slave-owners” (which seems to legitimate and rationalize the ownership of human beings), he uses “enslavers.” Small changes with big implications. These far more accurate and appropriate terms serve his argument well, as he re-examines the role of unfree labor in the rise of the United States as an economic powerhouse and its place in the global economy. In order to tear down old myths, he eschews the old language.

@absurdistwords had a great thread on this recently, urging us to “stop obscuring the horror with detached, antiquated, euphemistic terms”.

Clear Language on Slavery:

Slaves = Hostages
Slave Owners = Human Traffickers
Slave Catchers = Police
Plantations = Death Camps
Mistresses = Rape Victims
Discipline = Torture/Murder
Overseers = Torturers
Trading = Kidnapping
Profit = Theft
Middle Passage = Genocide

For example:

“The prominent slave owner never publicly recognized the offspring of he and one of his slave romances but allowed him to serve in the house”

is really

“The rich human trafficker raped his female hostage and then held their son hostage as well at the death camp he owned”

And from an earlier thread:

When you replace

“Owned slaves” with

“Was an active and willing participant in a vast conspiracy to kidnap children from their families in order to force them into industrial and sexual servitude”

It becomes harder to write slave owning off as just a blot on one’s record.

For instance:

George Washington was our first President and was an active and willing participant in a vast conspiracy to kidnap children from their families in order to force them into industrial and sexual servitude

They continue:

America treats slavery like an oopsie rather than a centuries-long campaign of nightmarish, brutal terrorism.

America sees the systemic and sadistic destruction of Black families as an etiquette violation.

Which is why it will excuse slave owners so readily.

Tags: USA   language   racism   slavery
Read the whole story
25 days ago
Boston, MA
Share this story

Life on Venus? Astronomers See Phosphine Signal in Its Clouds

1 Share

On such worlds, “as far as we can tell, only life can make phosphine,” Dr. Sousa-Silva said. She has long studied the gas, on the theory that finding it being emitted from rocky planets that orbit distant stars could be proof that life exists elsewhere in the Milky Way.

Here on Earth, phosphine is found in our intestines, in the feces of badgers and penguins, and in some deep sea worms, as well as other biological environments associated with anaerobic organisms. It is also extremely poisonous. Militaries have employed it for chemical warfare, and it is used as a fumigant on farms. On the TV show “Breaking Bad,” the main character, Walter White, makes it to kill two rivals.

But scientists have yet to explain how Earth microbes make it.

“There’s not a lot of understanding of where it’s coming from, how it forms, things like that,” said Matthew Pasek, a geoscientist at the University of South Florida in Tampa. “We’ve seen it associated with where microbes are at, but we have not seen a microbe do it, which is a subtle difference, but an important one.”

Dr. Sousa-Silva was surprised when Dr. Greaves said that she had detected phosphine.

“That moment plays in my mind a lot, because I took a few minutes to consider what was happening,” she said.

If there really was phosphine on Venus, she believed there could be no other obvious explanation than anaerobic life.

“What we find circumstantially also makes complete sense with what we know thermodynamically,” she said.

The team needed a more powerful telescope, and the scientists next used the Atacama Large Millimeter Array, in Chile, in March 2019.

Read the whole story
36 days ago
Boston, MA
Share this story

The Intercept Promised to Reveal Everything. But It Didn't Protect a Source.

1 Share

The startling carelessness about protecting Ms. Winner was particularly mystifying at an organization that had been founded on security. The Intercept had hired leaders in digital security, Ms. Clark and Micah Lee, for just such situations. Mr. Cole did not involve them at all.

Mr. Cole and Mr. Esposito said they’d been pushed to rush the story to publication, but Mr. Cole also acknowledged that failing to consult with the security team was a “face plant.”

The Intercept’s leaders argued in 2017, and still contend, that the narrative laid out by the Justice Department in its prosecution of Ms. Winner was shaped to make The Intercept — a thorn in the government’s side — look bad. And Ms. Winner’s own carelessness — she printed the document at work — could easily have gotten her caught even if The Intercept had been more cautious. But they also knew they had made real journalistic errors.

And so a key question was who to blame for this catastrophe and what consequences they should suffer. Ms. Dombek, who undertook the internal investigation, concluded that the editors — Ms. Reed and Mr. Hodge — needed to take responsibility. Others, including Mr. Greenwald, were demanding that Mr. Cole and Ms. Reed be fired, and The Intercept provide a public reckoning. (Mr. Greenwald later relented, and said he understood the desire not to “scapegoat” for an institutional failure.)

On July 11, 2017, Ms. Reed published a post on The Intercept announcing that First Look would pay for Ms. Winner’s legal defense. Ms. Reed also announced that an “internal review of the reporting of this story has now been completed.”

“We should have taken greater precautions to protect the identity of a source who was anonymous even to us,” she wrote. “As the editor in chief, I take responsibility for this failure, and for making sure that the internal newsroom issues that contributed to it are resolved.”

But the drama didn’t end there.

Mr. Greenwald and Jeremy Scahill, an investigative reporter who is the third founder of The Intercept, publicly demanded a more thorough investigation, and in response to their pressure, the company commissioned a second internal report, by a First Look lawyer, David Bralow. Mr. Bralow’s report, issued four months later, cited as central issues the decision to share the document with the N.S.A., Mr. Cole’s discussion of the postmark and the publication of the identifying markings.

Read the whole story
36 days ago
Boston, MA
Share this story

What Windows 95 Changed

1 Share
What Windows 95 Changed

Twenty five years ago today, Microsoft released Windows 95. It was undoubtedly a technical leap forward, but its biggest, most lasting impacts are about how it changed popular culture's relationship to technology.

For context, when Windows 95 was released in August of 1995, only about 30% of American homes had any computer at all. Less than 10% had any form of internet access — and virtually none had broadband. There were no smartphones, of course.

But more broadly, computers and software were basically not yet something one talked about in polite company. You might have had a friend who “worked in computers” (we didn’t say “work in tech” yet) or call IT for support for your printer at work. But software was not part of culture, and the term "apps" wouldn't come into wide usage for more than another decade. In those days, most job listings didn’t even yet ask for “familiarity with MS Office” (ask your parents what that meant) and the PlayStation hadn’t been released yet in the U.S. or Europe.

The broader business world had started paying a lot more attention to tech just a few weeks before Windows 95 arrived, when Netscape's milestone IPO in early August of 1995 shocked everyone with its extraordinary debut, and kicked off the dot com boom to come. But consumer marketing of PC technology was in its infancy; Intel had just named the Pentium not long before — before that, its chips were just referred to by their model numbers, which read like the license plate on a car, not a brand name. And even the Pentium name really only became famous when a bug was found in the early chips. Jokes about that were as far as pop culture really engaged with tech.

Into that world, Microsoft did a mass consumer launch of… an operating system. A computer's operating system is software that lets other software do interesting things. It's perhaps most abstract product possible. And Microsoft famously put some real money into it — they did a big launch event in Redmond and got Jay Leno to host it, and even licensed the Rolling Stones’ “Start Me Up” as a theme song, tied to the operating system's signature Start button feature. A lot of retrospective views of Windows 95 tend to focus on the kitsch value of Jennifer Aniston and Matthew Perry doing a VHS training video about the product, but the truth is, Friends was not a huge hit yet at the time (let alone a cultural phenomenon) and the video was a fairly obscure release at the time.

But splashy, big-budget ads for Windows 95 were ubiquitous in primetime TV in that era when everyone was still watching TV with ads. And as Brad Silverberg (then the SVP in charge of the Windows 95 launch) notes, "The Stones tv ad was so great because it was a kickass song, completely on message (Start), and showed everyday people around the world doing everyday things with the Win95 PCs."

Wildly, Microsoft's effort worked. For the first time ever, lots of consumer enthusiasts lined up to buy software at stores as it released at midnight. Before that point, that kind of widespread consumer enthusiasm had been limited to album releases and movie opening. Now, technology was part of the new world of fan culture. That fan enthusiasm had been slowly building for almost 2 years by the time of launch. Trade magazines had been writing about “Chicago”, the codename for the product, and geeks tried out the early public betas of the operating system — helping popularize the idea of a “beta” as a pre-final version of something in common usage.

There were even the early hints of toxic fandom that we're wildly familiar with now. Windows enthusiasts were sometimes oddly exuberant advocates for their preferred operating system, diehard fans of Apple (which was then a small player in a very precarious position) felt the new operating system ripped off their favorite OS, and partisans of IBM's offering called OS/2 Warp were the bane of every tech writer of the time, complaining about their favorite software being overlooked with all the fervor and indignation of today's most angry online comic book movie fans.

Despite that noise in the market, Windows 95 was inarguably a hit. And it changed how the rest of the tech industry worked. Modern tech culture and tech trade press still basically follow the conventions that developed back then. Reporters breathlessly cover new codenames and rumors and beta releases, and late night TV hosts don’t just joke about apps, they deliver their shows through them. When Apple talks about new version of its operating system as part of its annual events, they have all the production values of a high-end TV show.

And as a product, Windows 95 itself was fine. The user interface and design were certainly a leap forward over previous generations. There were decided user benefits in making it easier to configure computers, and it set the stage for later innovations where a normal person could plug in a mouse or keyboard into their computer and it would probably work. But the most lasting impact is how it changed the broader cultural perception of technology.

In the 80s, there had been a movie series “Revenge of the Nerds”; its last sequel came out only a year before Windows 95 did. The public perception of Microsoft founder and figurehead Bill Gates was as a caricature almost perfectly defined by the nerds seen in movie and on TV, only missing the tape on his broken glasses. Tech was seen as for those people — nerds who were walking punchlines.

But after Windows 95 arrived, tech quickly became a standard part of people’s lives. The Internet became mainstream, homes got connected, and software became something everyone uses. Eventually, smartphones put a computer in everyone's pocket, not just in their homes, and software became "apps" — and became part of our lives.

Operating systems went from a product that we buy to a fundamental capability that's bundled with the entire tech ecosystems where we live our lives. We don't pay for operating systems directly anymore by purchasing them, but instead we pay with surveillance of our data or by being sold connected cloud services or by the cost being bundled into our devices. Operating systems are both ubiquitous and invisible, and there are now people for whom their allegiance to the operating system of their phone or video game console or even personal computer is part of their identity.

And the Start button is still pretty cool.

Read the whole story
40 days ago
Boston, MA
Share this story
Next Page of Stories