GitHub Service Outage (github.com) 117
New submitter thebigjeff writes: Beginning at around 7:30pm EST on 1/27/2016, GitHub's core services have been offline. Most repositories and other functionality is inaccessible. The status page is calling it a "significant network disruption."
More from The Register: GitHub falls offline, devs worldwide declare today a snow day.
"7:30 PM" (Score:1, Insightful)
So is that Ulaan Bataar Standard Time, or what?
This is almost as bad as the suits in California thinking that *of course* all its employees in Europe and Asia use PST. Gawdz.
Re: (Score:1)
Re: "7:30 PM" (Score:3)
Still they are an international service with customers around the world. Therefore, it would be logical to usw UTC. Especially, as most people know their difference to UTC, but not some freaky timezone in a country far way. Alternatively they could use AOE , but that is only known by scientists as the paper deadline time zone.
Re: (Score:2)
Hey Watson! There are several time zones is US.
P.S. Everybody is used to UTC to avoid fuckups in major big iron stuff.
Re: (Score:2)
Which would be equally unhelpful for readers that are not familiar with UTC
Fortunately, for now, most readers of Slashdot are from Earth. Every time zone is expressed as an offset from UTC. Given a time in UTC, it is easy to work out what it is in your own time zone. Given a time in any other time zone, you do the conversion by first looking up the UTC offset of the reported time zone, subtracting it, and then adding your own UTC offset.
Re: (Score:3)
Re: "7:30 PM" (Score:2)
Europeans don't put "Standard" in their time zone, because that's meaningless.
Re: "7:30 PM" (Score:1)
"Standard" is super meaningful.
EST = Eastern standard time. New York is on this right now, but when daylight savings begins, it won't be.
EDT = Eastern Daylight Time. This is used in America during the summer on the east coast.
Re: (Score:2)
Re: "7:30 PM" (Score:3)
Right, and that's extremely confusing because elsewhere in the world S means Summer.
WET/WEST, GMT/BST, CET/CEST, etc.
There is nothing "standard" about a time zone, so the deduction that S refers to summer time makes more sense.
Re: (Score:2)
Easter Standard time aka America New York Time which is called EDT (Eastern Daylight Time) in summer UTC-5/UTC-4
Re: (Score:2)
Except that at the moment, they're not on DST (EDT), they're on EST.
UTC is useful, but I'll settle for the name of the locale. It's not like people don't know where New York is.
Re: (Score:2)
Updated with correct time zone (EST). Good catch.
Who are you and what did you do with the /. editors* ????
*And by editors I mean badly written Perl scripts that don't even have a spell check module installed.
Re: "7:30 PM" (Score:2)
also what is this 27th month they speak of?
Re: (Score:2)
This is a very common problem... I quite often get people online giving me their phone number without a country code, or their address without a country and in 99% of cases they are in the US when they do this. Often the last part of the address is a two letter code for their state (e.g. CA) which could easily be misinterpreted as a country code.
The Internet is a global network, when sending emails or posting data online you should absolutely declare the country if you're giving out a physical address or ph
Re: (Score:2)
Within North America, we don't use country codes, even when calling internationally. Calls to Canada or the Caribbean are just direct-dialed exactly as if they were in the U.S. Therefore, we don't think about country codes, even if we do transact cross-border business. I couldn't even tell you what the country code for the U.S. is, or if there is just one code for all NANP territories or if there are separate codes per territory. When we do call outside the NANP area, we have to prefix 011 to indicate this,
Re: (Score:1)
"Within North America, we don't use country codes, even when calling internationally"
Yes you do.
You've reinforced the parent poster's main comment that American's don't know shit about how things work in the rest of the world.
The USA and Canada have the *same* country code. So guess what? You don't need a code to call between them. This also applies to some US protectorates and the Caribbean, but not Mexico, that's +52. See below for a listing.
+1 COUNTRY CODES [wikipedia.org]
+1 Canada
Re: (Score:2)
People use the US country code all the time and have for decades. It's the "1" in "1-800" numbers.
But since the US does so much intra-quadrasphere calling where an area code is sufficient, they don't realize it's a country code and just assume it's part of the "800" system.
Re: (Score:2)
+1 and 1 are not the same, just as +52 and 52 are not. We don't write +1-xxx-xxx-xxxx because that would mean actually dialing 011-1-xxx-xxx-xxxx. +52 really means 011-52. 1 was always billed as "the long distance flag" so to speak, until area codes became mandatory for local dialing as well due to overlays.
Re: (Score:2)
It's been many, many years since the +1 has been mandatory for a large portion of the US
It's still mandatory if you're calling from outside North America - same as everywhere else on the planet.
Re: (Score:2)
Within North America, we don't use country codes, even when calling internationally.
Good trick if you can manage it. What do you use - owls?
though in writing this is commonly represented simply as + (and people are just expected to know 011 is the replacement)
This is the standard way of representing "code to go international", the point being that it varies from country to country.
I can give my telephone number as +44 1491 NNNNNN
and then anywhere in the world, anyone knows to dial their international access code (in the UK it's 00; it used to be 010), followed by the rest of the number.
FYI, the international code for North America is "1".
Re: (Score:3)
FYI, the international code for North America is "1".
So when people say America is #1, they are just static a well known fact.
Re: (Score:2)
Yay, autocorrect.. stating a well known fact.
Re: (Score:2)
Within North America, we don't use country codes, even when calling internationally.
Good trick if you can manage it. What do you use - owls?
We use area codes, and within NANP territory, these are country-agnostic. This is convenient in some ways, but also disastrous in others because someone can give me a number in the Bahamas where I'll get charged $2.00 a minute for the call, and it won't look any different from a call to another state or even within the same state. There is absolutely no rhyme or reason to area code assignments *except* for the Caribbean, where they were generated base
Re: (Score:2)
So you're familiar with the postal code system in every country in the world?
Sometimes slashdot is slow (Score:2)
but it wasn't long ago (1 hour) since I found out about the outage, by seeing an unicorn. Congrats slashdot for being fast.
Re: (Score:1, Funny)
Probably a side effect of the Code of Conduct. Some SJW admin decided to sed -i s/manual/femual/g and it brought the whole thing down.
Must be the OS/2 Source Code (Score:3)
MMMMMM.. how about that mother fucking cloud? (Score:5, Funny)
maybe they should've backed up their cloud in another cloud. Cloud.
Re: (Score:2)
I hear Google Actual Cloud is good.
Re: (Score:1)
This cloud precipitated a failure.
Re: (Score:2)
They seemed to have done so for their billing platform which had no problem charging me last night at 7:58pm
Back up now (Score:2)
did github backup (Score:2)
Re: (Score:1)
Decentralized source control (Score:5, Insightful)
The selling point of git was to be a decentralized source control system.
It is interesting to see people telling about a snow day while they have a tool that do not require a central repository
Re: (Score:2)
I came here to say exactly this. Then I realised that there's more to github, such as the issue trackers etc, than just git itself.
Re: Decentralized source control (Score:4, Insightful)
There are also decentralized ticket systems. Even though you should know enough about your tickets and tasks to have something to do for one day ;-) And if all fails, update the documentation.
Re: Decentralized source control (Score:5, Funny)
And if all fails, update the documentation.
Github mostly hosts open source projects.
</snark>
Re: (Score:2)
True. however, good OSS projects document in the code describing the intent of a module or provide text files with the code (for example the Linux kernel). Others ignore documentation and then they wonder why they cannot really attract new programmers. Anyway, the beauty of git is, you can work offline and still be productive or counter productive (depends on your writing skills).
BTW: Commercial software also has often no documentation, documentation which describes what the code does (which should be obvio
Re:Decentralized source control (Score:5, Insightful)
Even if you are using a source control system that *does* require a central repository to be up: since when did the inability to check in prevent you from writing and debugging code? If interacting with git/svn/clearcase/etc. is more than 0.1% of your work day, maybe you're not doing it right.
If source control being inaccessible means you get the day off.... let's just say that ClearCase users would be extremely happy.
Re: (Score:3)
Github is much much more than version control. It's also bug tracking, feature tracking, discussions, web hosting, wiki, release management, etc. When all that goes down, you can still write code, but you can't communicate with the other devs anymore.
Re:Decentralized source control (Score:5, Insightful)
So: no incoming bugs or feature requests, no merging other people's code, nobody pinging you every 5 minutes? Around here that's called "a day where I can be productive".
Re: (Score:2)
Don't you have pull requests that need your attention?
Maybe there are pull requests that required attention before the outage, and that are in the works. Maybe you cannot commit, but you have more test cases to refine, more documentation to write. Oh yes, the stuff you have on your backlog, you can work on that too while you are at it. And if you really have to have someone get your changes (because shit, they are urgent), you can pull a worst-case scenario and e-mail your changed files to the appropriate recipient (because I'm certain your e-mail is not depe
Re: (Score:2)
No ability to use the automated build system that pulls updates or source code exports from git tags at github. No configuration publication or web content updates with github based branches. Sharing code between repositories locally is still feasible, but loses the insurance that the code submitted to production has been submitted somewhere accessible to other programmers.
Re: (Score:2)
No ability to use the automated build system that pulls updates or source code exports from git tags at github. No configuration publication or web content updates with github based branches. Sharing code between repositories locally is still feasible, but loses the insurance that the code submitted to production has been submitted somewhere accessible to other programmers.
Email the deltas if you have to. And if you are in a real emergency, you can clone and upload your local copy into bitbucket.
We had a situation like that where we lost our infrastructure a couple of months ago. We couldn't code, we couldn't build, we couldn't do integration testing. Total blackout. Rather than waiting for Ops to bring everything back online, we stopped coding and migrated everything we needed on a different system. We lost a lot of history, but we were back on track.
It was either that
Re: (Score:2)
That's why I have an office I can go to. You know, for all the working with coworkers,
Dude, just, no. (Score:2)
Github is much much more than version control. It's also bug tracking, feature tracking, discussions, web hosting, wiki, release management, etc.
Not enough for a snow day (unless you are doing it wrong.) Hell, if you are doing it right, you can still be productive during a days-long outage.
When all that goes down, you can still write code, but you can't communicate with the other devs anymore.
Email, IM, skype. I mean, Jebuz on a pony, you make it sound like there a civilization collapse, and that we start using smoke signals and runners carrying clay tables filled with cuneiform.
Re: (Score:2)
If source control being inaccessible means you get the day off.... let's just say that ClearCase users would be extremely happy.
As someone who used to be a ClearCase/MultiSite admin in a former life, I have to say that if that system went fully offline for anything more than a few minutes, then your admin isn't doing it right.
Re: (Score:2)
As a user of source control in general- if you need an admin for it, you're doing it wrong.
Re: (Score:2)
I've worked with source control since RCS very shortly after RCS was first published in 1982. If you don't have an admin, to at least ensure backups, consistent merging practices, and cleanups when someone stores huge binaries accidentally, your source control is in real danger. I'm afraid those are all typically human usage issues that require at least a slice of someone's time.
Debugging corrupt content on the back end of the service is its own issue. It's happened with every major source control system I
Re: (Score:2)
You need an admin for the server, who's also admin for the other boxes you own. You don't need a specialized admin for source control, like ClearCase did (in fact medium sized installations generally had a small team of admins, and the fucker still sucked preformance wise. RCS was less annoying to use, much less anything newer).
Re: (Score:2)
Oh, I see. You meant "if you need a dedicated admin whose sole task is source control, you're doing it wrong". Yes, I'd agree with that.
Re: (Score:2)
As a user of source control in general- if you need an admin for it, you're doing it wrong.
No. You are doing it wrong (inexcusable) , or you are not working on a large scale system (understandable.). As your systems and teams grow in size and complexity, you need gatekeepers. And you need people in charge of doing sysadmin work, backups and stuff, including maintaining and backing up your main repositories.
Beyond a certain team size, it is not cost effective to have developers managing those resources. You want them to develop. Yes, you might have a few developers part-timing on those roles (o
Re: (Score:2)
It makes sense to have the boxes all under 1 team, sure. But source control is a pretty fucking simple concept. If you need a team (or even 1 full time admin) to keep it running, your system is fucked up (hi ClearCase users). It should be yet another box owned by IT, not a full time source control admin.
Re: Decentralized source control (Score:2)
Say you're on a team of 10-20 devs in a mature product. Your day job is to work through the issue backlog, hopefully checking in fixes for three or so issues a day. You need to pick issues from the issue tracker (GitHub), read comments, maybe interact with the filer to get repros, submit a PR for code review by your peers (on GitHub), send it off for regression testing (via guthub messages), code review about 5 other fixes from other team members (GitHub). Also triage incoming issues (github). That's a heck
Re:Decentralized source control (Score:4, Insightful)
I know it's fun to be snarky about the fallibility of the cloud at times like these, but in fairness, I think one has to measure these unexpected outages against the productivity gains of having a convenient centralized point to synchronize your project online, especially for historically decentralized teams like your typical open source projects.
The notion that "git is decentralized" is obviously tempered against the requirement to synchronize everyone's repositories, right? Still, I agree... the whole "github is down, I can't code today" is an even weaker excuse than something like "it's okay if I'm goofing off - I'm compiling." [xkcd.com] One of the benefits of git (and Mercurial as well, which is actually my system of choice) is that it's trivial to make a local branch and start working on some new feature. If you're working on a project, then by definition you have an entire copy of the repository locally - it's not like you need to connect to github just to see your code or check in changes locally. Even if you can't see your bug/todo list, that just means it's a great time to make a branch and start some other little project, like doing some refactoring or code cleanup - or even, heaven forbid, some documentation.
Re: (Score:3)
Not all development work involves solitary coding. How do you get the latest changes from a co-worker when you can't access the repository you both normally push changes to and his personal machine with his copy of the repository doesn't accept incoming connections (and neither does yours, so you can't have him push the changes to you)? How do you access the branch you didn't know you needed until now which isn't in your local copy? How do you get that refactoring a colleague just committed and pushed befor
Re: (Score:2)
Or , you could export the patches, move them to another machine, and import them. Exacly what would happen automatically wiht push/pull. But it still will have the same information as far as changesets/etc. so it'll seemlessly work once its up.
Re: (Score:2)
Isn't that why the enterprise version of GitHub exists for locally hosting the service? Isn't that how GitHub makes money and subsidizes the free services?
And you're putting up some pretty big barriers by saying neither user's box allows network shares or an sshd...
Now one big issue with github going down isn't it stopping programmers from writing code but preventing some people from deploying code.
Node.js npm and Rust crates package managers sometimes point to github repos for packages...
Re: (Score:3)
Or you send patches by e-mail, just like Linus did himself before start coding git.
Re: (Score:2)
Linus and the kernel developers had been using Bitkeeper for free when there was a licensing problem. _That_ was when Linus wrote git, to effectively use the style of good quality merges that Bitkeeper previously provided, in a free software way.
Re: (Score:2)
"Linus and the kernel developers had been using Bitkeeper for free when there was a licensing problem."
There is a difference between "before" and "immediately before" you seem not to grasp.
Re:Decentralized source control (Score:4, Informative)
It is decentralised, I was still able to commit changes to my repo and carry on as normal. What I couldnt do was use GitHub.
GitHub has value adds which make it a nice thing to use - its an off site repo for backup, it has a nice PR and issue handling system, it has nice metrics, it has commit hooks, it acts as a good point for CI service to integrate with automatically (alternatives being you either have to handle CI locally, manually push changes to a CI repo, or expose a git repo somehow so a CI service can grab checkins and build them).
So I couldn't push my changes to GitHub and my CI service didn't run new commits for a few hours. Not to worry, its already caught up with the back log.
Do not confuse GitHub with git - the two are entirely different. GitHub could use CVS and still have all the value adds, it would just use a shit source code management system.
Re: (Score:2)
If you have a clue, that's true.
99.999% of "developers" only know that you click commit and push in SourceTree ... They don't actually have any clue what that does.
Re: (Score:2)
Shhhh......you're ruining another excuse to swordfight in rolly chairs.
http://xkcd.com/303/ [xkcd.com]
Timespan (Score:2)
It was between 7:22 and 7:24pm EST when it went down judging by my last commit and the next attempted commit
Not everyone (Score:4, Interesting)
Some of us use BitBucket you know... no interruptions for me today!
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
I've been bitten by more BitBucket outages than I've seen GitHub disruptions :p
Which is why you should have code bases in both (pick your primary in either, and keep the other one as a hot backup.) If a system is worth going through the trouble of constant availability and reliability, this is the only way to go.
Re: (Score:2)
Why? Because it's down SO often you just stopped using RCS all together?
No, but you'll have one tomorrow, 2 on Saturday, ssh won't work most of Sunday ...
Don't try to pretend BitBucket is reliable. It is hands down the most unreliable service I deal with on a daily basis.
Our company is currently split between BitBucket for private Repos and GitHub for public ... But only for a few more weeks as we finish abandoning the crapfest half assed mess known as BitBucket
Re: (Score:2)
Sure, unless today is the end of a sprint and you have 20 developers trying to merge their feature branches into the deployed branch.
Re: (Score:1)
A proper scrum master would never have such problems.
Re: (Score:2)
A proper scrum master would never have such problems.
Wait, scrum masters do source control administration? :)
Re:How is it a snow-day if you're using Git? (Score:4, Funny)
it just means you make a few more commits before pushing and pulling
Funny, I got the same suggestions in health class in high school...
Remember that Github wikis are also repos (Score:2)
Project Late (Score:2)
Somebody's project was late and needed an excuse so they crashed github.
Re: Fuck GitHub! (Score:1)
I find your views intriguing and would like to subscribe to your newsletter.
Welcome to the cloud! (Score:1)