What America's NSA Thinks of Python (zdnet.com) 74
"Now budding Python developers can read up on the National Security Agency's own Python training materials," reports ZDNet:
Software engineer Chris Swenson filed a Freedom of Information Act request with the NSA for access to its Python training materials and received a lightly redacted 400-page printout of the agency's COMP 3321 Python training course. Swenson has since scanned the documents, ran OCR on the text to make it searchable, and hosted it on Digital Oceans Spaces. The material has also been uploaded to the Internet Archive...
"If you don't know any programming languages yet, Python is a good place to start. If you already know a different language, it's easy to pick Python on the side. Python isn't entirely free of frustration and confusion, but hopefully you can avoid those parts until long after you get some good use out of Python," writes the NSA...
Swenson told ZDNet that it was "mostly just curiosity" that motivated him to ask the NSA about its Python training material. He also said the NSA had excluded some course material, but that he'll keep trying to get more from the agency... Python developer Kushal Das has pulled out some interesting details from the material. He found that the NSA has an internal Python package index, that its GitLab instance is gitlab.coi.nsa.ic.gov, and that it has a Jupyter gallery that runs over HTTPS. NSA also offers git installation instructions for CentOS, Red Hat Enterprise Linux, Ubuntu, and Windows, but not Debian.
"If you don't know any programming languages yet, Python is a good place to start. If you already know a different language, it's easy to pick Python on the side. Python isn't entirely free of frustration and confusion, but hopefully you can avoid those parts until long after you get some good use out of Python," writes the NSA...
Swenson told ZDNet that it was "mostly just curiosity" that motivated him to ask the NSA about its Python training material. He also said the NSA had excluded some course material, but that he'll keep trying to get more from the agency... Python developer Kushal Das has pulled out some interesting details from the material. He found that the NSA has an internal Python package index, that its GitLab instance is gitlab.coi.nsa.ic.gov, and that it has a Jupyter gallery that runs over HTTPS. NSA also offers git installation instructions for CentOS, Red Hat Enterprise Linux, Ubuntu, and Windows, but not Debian.
public domain (Score:3, Insightful)
Re: (Score:2)
Their bugs are in the public domain. Just break open a motherboard. :-D
Re: (Score:3)
I wish that government bodies put these kind of texts in the public domain.
That is not going to happen.
When the government doesn't like citizens being uppity, they will find subtle ways to get even. One way to do so is to take electronic documents (such as PDFs), print them out, then scan them back in as rasterized images, and then provide the citizen with the information in a bloated and far less useful format. That is what the NSA is doing here, so they clearly don't like giving out this information.
Government prosecutors use the same print-scan-send technique when giving disc
Re: (Score:2)
This tends to be a side effect of the redaction process, in an effort to makes sure redacted text is not recoverable.
Re: (Score:2)
Re: (Score:2)
FTP? You're a funny old man. That's been obsolete for decades due to unmanageable security flaws.
Re: (Score:2)
R has already replaced it for a lot of people.
Re: budding Python developers (Score:2)
I thought that was PHP...
Re: budding Python developers (Score:5, Funny)
PHP is the hot dog stand with the pealing yellowed cease and desist order from the health inspector of programming.
Re: (Score:1, Informative)
I heard people shittalking perl like that in the 90s and 00s, while I was paid six-figure salary to make things with it.
Then two companies which stock I had IPO'd.
You blokes still have to work, don't you?
Development code not necessarily one day's effort (Score:4, Insightful)
Exactly. Python is the fast food of development tools. Good for a one time task that you'll just put in the shitter the next day.
Not quite. Certainly python is abused, used in inappropriate situations, but it has its uses. For example gluing together calls to the OpenCV computer vision library. All the heavy work is done in the library. Python however is more convenient for gluing together the library calls during development as you explore different algorithms with different parameters. Once all that is figured out I'll probably glue the calls together in whatever the native language of the platform is; Swift for macOS and iOS, java for Android and AWS, C# for Windows, etc since I'll probably be touching platform specific UI code to show work in progress and/or results. And if I wrote any custom image twiddling code during development I'll rewrite that at some point in C/C++ w/ Intel's portable SIMD extensions.
;-)
In short python possibly during development but probably not in a shipping app. But I am not talking one day's worth of disposable code, I'm talking development that is probably measured in weeks or months.
Another example would be in academic research, no one gives a crap about a fancy end user app. Its all about the data in and the data out. So for this audience your development python gluing together things is your "production code", your "deliverable".
Python is just a language, one of many, and has good fits and bad fits like all the others. The mistake is thinking there is one language solution for everything. Yes python is abused in that manner by some but so is pretty much every language. Its OK to have a favorite or preferred language but you have to know a few to have the flexibility to make reasonable choices. Python would be a tool in your toolbox. I wished I learned it earlier, I would have used perl a lot less.
Re: (Score:2)
Re: (Score:1)
Re: (Score:1)
Vast amounts of plain text data needs to be sorted as the NSA collects all.
Its "OCR" on the "text" to make it "searchable" that needs so many people.
The "how" and "why" of the NSA still getting easy to work with "text" in such an encrypted world is not mentioned
People who can be trusted not tell people/gov/mil of another nation, faith/criminal/banned groups about their OCR text work at the NSA.
Not to talk to t
I Fully Agree with the NSA View (Score:5, Insightful)
I have been coding since the mi-1980's, starting with BASIC, Assembly, then C, then C++.. and on through many other languages today being mostly JavaScript (browser-based and Node.js), C#, Python, and SQL.
Python always feels to me like a "baby language". That's ok because it does serve its niches reasonably well. It's easy to learn, easy to read, and it has libraries for working with most kinds of things that it would be too slow for, such as processing large lists of things. It certainly also has its ugly parts, also.
Using libraries, it's ok (not great) for data processing (requiring high performance and flexibility over large data sets) and is actually quite good for business logic (requiring simplification and clarity).
However, there hasn't been a decent language built for data processing or business since COBOL. SQL quickly becomes insanely complicated when evaluating problems that span multiple inter-related rows. Graph databases handily solve nearly every problem one can imagine, particularly Neo5j, but the query languages/methods themselves add complexity. Other NoSQL databases also tend to lack in expressiveness and performance with regard to flexible analysis. For example, MongoDB is fine if you keep all your data organized in specific documents and don't anticipate any substantial complexity between them. Their developer's comment that you should "do your joins at design time" illustrates that they just completely don't get it. Also, the JSON-based query language is hard to look at, even with simple queries.. MongoDB really is for front-end developers of simpler, isolated or non-enterprise applications. It will not do for data warehousing or sophisticated report capabilities, such as from data cubes, etc.
I actually did some work using Microsoft JScript that really impressed me for data processing and business logic. JavaScript data structures are quite versatile, fast, easy, and clear to work with (once a new user understands pointers). And in Microsoft JScript, you have synchronous access to databases like SQL Server. Although this does not perform nearly as well as Node.js, it provides for the ability to specify synchronous business processes and abstract them into sub-functions. You cannot do this in Node.js. The performance hit to execution of JScript is not too much in exchange for the vastly greater developer performance and reduced risk of coding mistakes. There are no race conditions. A completely lack of synchronous database access is the only thing holding back Node.js from being the overall best data processing and business logic language of all time, in my opinion.
Actually, I wrote my own Tree Graph database that uses the same data object access syntax as does JavaScript. I simply extended it a little. Mainly, I enabled query notation with square brackets. For example,
myorg.projects[ department = "mydept" and budget.projection > budget.authorized ].name,manager,budget.projection-budget.authorized;
Currently, I am working (in what spare time I have) to fully integrate this with my p2p mesh network. A dream, if I had the time, would be to fork Node.js such that this is its native memory system. In this case, I would enable persistence to conditions for data arrays and objects, the default condition being normal JavaScript variable scoping. Otherwise, they would be persistently stored in the mesh. So for example, you could specify a certain data object to persist only while a certain process is running or until some event occurs.
The mesh "charges" for use of computational resources. One's computer tied to the mesh provides computation resources, like memory, disk storage, computing time, bandwidth, and anything else custom (e.g. interface to human user or interface to static IP address). So you get "paid" for what you provide to others and you can use those credits to use the resources of others, in exchange. Prices are strictly and mechanically supply and demand with also some taken for mesh overhead.
I talk too much.. sorry.. It's sunday morning and I need to get back to work on that.. it's what I am working on now... other than slashdot.
Thanks,
--Matthew C. Tedder
Common table expressions (CTE) solve that (Score:3)
> SQL quickly becomes insanely complicated when evaluating problems that span multiple inter-related rows.
Spend an hour learning common table expressions and try using them next time. They largely solve that. They are similar to functions in other languages.
Without CTEs you can have statements embedded inside statements embedded in statements six levels deep, like this:
Query1 join
Query2 Where
Query3 join
Re: (Score:2)
"Python is the closest thing there is to Lego in programming languages."
Squeak
Re: (Score:1)
"Python is the closest thing there is to Lego in programming languages."
Squeak
Just do not step on it when the lights are out.
But it IS good some things - big ints, one-time us (Score:2)
Having said I wouldn't generally build enterprise-grade systems with Python, I should acknowledge where I do see it as useful.
It can be great for a script you are going to run once, or little utilities for use on your personal desktop. For these, you don't invest in the kind of practices you use for production systems, peer review and all of that. Quick and easy fits here.
Python also happens to be good with large integers. I've used it to break encryption, where you are dealing with 128 bit and larger nu
Re: (Score:2)
You CAN do the same with temporary tables, but then you've started doing imperative scripting in a declarative language. That will ruin performance as well as screwing up your ability to think declaratively.
If you have a query to fill a temp table transforming it to a WITH clause or the other way around is child's play.
INSERT INTO #temp1 SELECT [long ugly query]
INSERT INTO #temp2 SELECT [query with temp1]
SELECT * FROM #temp2
vs
WITH [long ugly query] AS cte1, [query with cte1] AS cte2
SELECT * FROM cte2
One enforces an order of execution and the other doesn't, but all the hard bits of figuring out if the query actually returns what you want is the same. My experience has been the opposite of yours, you can throw e
Re: (Score:2)
True, you absolutely CAN partially fix your performance problems (temporarily) by throwing out the entire concept declarative programming and writing a imperative script instead. That works okay until something changes. Of course, that brings up the question of why you're using SQL, if you want to write imperative scripts. It's kinda like finding a way to haul bricks on a motorcycle at that point.
It's also a very sub-optimal solution because temp tables themselves are dog slow - just about the slowest op
@expensive_parts more clear than expensive_parts ? (Score:2)
> it's like showing your work in math not just the final answer on the dotted line
If you think that through, you're suggesting that this:
SELECT * FROM @managers
Is easier to read than this:
SELECT * FROM managers
You really think the @ sign makes it a lot more readable?
That's the difference between using a CTE and a temp table. The @ sign, which forces the server to run that part of the query NOW, and write all of rhe results disk - even when it's not going to use half the results because there is an inde
Re: (Score:2)
Another great thing about SQL is that, like COBOL, you have convenient access to decimal math, the stuff that humans use.
Re: (Score:2)
What programming language are you thinking of (besides assembler) where you can't do decimal math?
That said, I like Tzeltal math. Base 20. Those Mayans really knew what they were doing.
Re: (Score:2)
No, Mr. Dumb Ass, writing your own math routines is not a serious answer.
Go and do somebody's accounting in base 20 and see if you get the same number of bugs as somebody using decimal math. And see if the code auditor gives you high marks for being clever.
Your attitude is the exact type and character of stupidity that leads to it being a good idea to do accounting in COBOL. Anything other than decimal math is too hard in COBOL, so you won't screw the accounts up with a float.
Programming having to do with k
Re: (Score:1)
Re:I Fully Agree with the NSA View (Score:5, Insightful)
Examples please? I have done many data processing programs that hold a few (5-16) GBs of data in RAM at a time over the years (mid-90s). I am not talking about RAW data, but just the information that needs analysis.
I honestly haven't found anything that comes close to the flexibility, turnaround, and dev speed as Python or Perl. Yes, C, C++, C#, Java, etc do the analysis faster (20% for C) and I can crunch that 5GB down to 3GB with proper C structs, but loss in overall time & speed for changes put the Ps in front. Of course the data processing isn't pure Python, the heavy number crunching is entirely C (5% of the process) and many of the analysis libraries also have their heavies in C/C++, but overall... you get the dev speed and flexiblity of Python with barely touching C.
There are also other inefficients that I add to provide clarity in the process overview & juggle system resources. Like temporary Sqlite instances, remote SQL servers, status/data messages between instances, etc. But Python just makes glueing all these various tools together so easy that it doesn't really add complexity but does allow various tools to do the part they are good at.
Re:I Fully Agree with the NSA View (Score:5, Insightful)
I have done many data processing programs that hold a few (5-16) GBs of data in RAM
The size of the data isn't the issue, it is the size of the program. Python is great for programs of less than a thousand lines, worked on by one person, and that have no serious consequences for a runtime error (nobody dies).
When you move out of that comfort zone, and into million line projects, developed by teams of people of varying levels of competence, and with serious reliability and performance requirements (plane shouldn't fall out of the sky), then Python is not the right tool.
Re: (Score:1)
Re: (Score:2)
I honestly haven't found anything that comes close to the flexibility, turnaround, and dev speed as Python or Perl.
Until you need to refactor or work with lots of people....
Re: I Fully Agree with the NSA View (Score:1)
Re: (Score:2)
Graph databases handily solve nearly every problem one can imagine, particularly Neo5j, but the query languages/methods themselves add complexity.
I'm with you there, graph databases are great. Their only drawback is that finding people who know how to work with them is hard.
is the only thing holding back Node.js from being the overall best data processing and business logic language of all time, in my opinion.
I don't see how you can get from COBOL to this. COBOL is better than Node.js.
"Python isn't entirely free of frustration ..." (Score:2)
Re: (Score:2)
The "designate a block of code by the amount of indentation" aspect is a pain to get used to.
Initially it is annoying, but you do get used to it within a few hours. Personally I believe it is the least of Python's problems, though it does appear to be the one that most people complain about.
Re: (Score:2)
The "designate a block of code by the amount of indentation" aspect is a pain to get used to.
Initially it is annoying, but you do get used to it within a few hours. Personally I believe it is the least of Python's problems, though it does appear to be the one that most people complain about.
That's because it's dumb and unnecessary; implemented as a whim by it's creator. Using actual delimiters to delimit blocks has no downside. I'd be fine with Python *allowing* whitespace-delimited blocks as long as they weren't required. I think it's a nice feature for short and/or logic (not syntax) learning purposes -- pseudo-code that runs. Longer, more complicated things benefit from more, not less, defined structure.
Re: (Score:2)
Re: (Score:1)
That's because it's dumb and unnecessary
No, that's because it's immediately obvious. It's an easy target without having to dig deeper. This is a benefit both in making the objection and in finding a large audience that shares that understanding. It's a far easier argument to make than any of the arguments in favor of Python.
Which I'm ok with because I think "one way to do it" limits creativity and hurts the future of software.
Re: (Score:2)
Re: (Score:2)
I agree, although I will note that nobody complains about Haskell's offside rule.
I think that the reason why people complain about it in Python and not in Haskell... well, there are several reasons, and one of them (tabs vs spaces) is stupid and belongs in the 90s.
Apart from the difficulty of refactoring, I expect that one of the big problems is that a lot of programmers coming to Python see indentation and assume it's a block-structured language like C or Perl, then get a rude shock when they find that it
Re: (Score:3)
The "designate a block of code by the amount of indentation" aspect is a pain to get used to.
There's a great workaround for that called Perl. :-)
Re: (Score:2)
Re: (Score:2)
> There's a great workaround for that called Perl.
Some people are easily confused by symbols outside of [A-Za-z0-9].
Re: (Score:2)
Now I guess I know how you old fogeys still slinging COBOL feel!
Re: (Score:2)
The "designate a block of code by the amount of indentation" aspect is a pain to get used to.
It is easy to get used to, but will never stop feeling stupid, because IT IS stupid ;)
but not Debian...pfft (Score:2)
You're saying the Ubuntu instructions wouldn't work on Debian?
Re: (Score:2, Interesting)
You are the worst kind of fool. You live in near perfect safety, with abundant food, shelter, luxuries that would make the Romans blush, but have no idea how that was attained and is maintained, and think *your trivial problems* are the worst kind of oppression You can spew any sort of idiocy you want, all day long, with no consequences or anybody even caring very much.
Groups like the NSA *are not the enemy*, they are *the reason you can make profoundly stupid statements in full public
Re: NSA ads on Slashdot... seriously? (Score:2)
Re: (Score:1)
the PRISM slides showed how.
The Church Committee https://en.wikipedia.org/wiki/... [wikipedia.org] hinted at why.
Re "*not one whit*"... a lot of time and effort went into the consumer OS collection efforts
If it was useless noise the NSA would not be collecting it all for decades.
The morons that get mod points get dumber and dumb (Score:2)
This is a literal idiocracy. Rule by idiots.
So typical of no-"edukayshun"-Murica.
As a teacher I looked at it (Score:2)
I used the Microsoft course in the past. It worked reasonably well. I did it as a"watch them do in the video, then watch me do, then go as a whole class. That worked well, but I had a lot of students transfer out right away, as they found it to
Re: (Score:3)
The culling is working pretty well, though not well enough. I would estimate that the rate of "programmers" existing in the general population is about 1 in 100,000 (0.001%), which is about 1,000 times the rate of the existance of other "it just works" specialists (mathematicians, musical composers, etc). So if you are a "good teacher" you might expect to come across *ONE* programmer in your entire lifetime. If you are lucky!
While any asshole can learn to code (and many do) very few are skilled at it. A
Re: (Score:2)
Re: (Score:2)
"There is a reason abstract algebra is not taught rigorously in primary school, and instead things like basic numeracy and counting are emphasized first."
As Homer Simpson would say, "Doh!". Before you can understand abstract algebra one must understand basic numeracy and counting. You would find it very difficult to teach the solving of linear algebra equations to someone who does not know what a number is, and what a variable is, even if they were 80 years old.
"Now it is the information age but that is n
Re: (Score:2)
Guess you're lamenting the fact that you're in the 99,999, eh?
Re: (Score:2)
Its a tricky problem - the best teaching language may not be the best production language. Python has a lot of "magic" things that are handled out of sight. There is some advantage to a simple language (some modern equivalent of BASIC) that lets students have a better feel for what the computer is actually doing.
It might also work to teach a limited version of Python.
Re: As a teacher I looked at it (Score:1)
Re: (Score:2)
One thing I am NOT to teach is Java (no real tears over that one). The reason is that the High School teaches it (it is still part of the AP curriculum). They want to start from zero.
Said in the light of the comment above, on the probability of a great programmer, I am not a great programmer. It just doesn't draw my interest. So, yes, it needs to be someth
for CentOS, Red Hat Enterprise Linux, Ubuntu, (Score:1)
CentOS, Red Hat Enterprise Linux, Ubuntu, and Windows, but not Debian, or TempleOS.