290
291

10 Myths of Enterprise Python (paypal-engineering.com)

pythonautical が 投稿

全ての 147 コメント
並べ替え:
ベスト
ベスト
トップ
新着
注目
論争中
古い順

[–]kylotan 13 ポイント14 ポイント  (12子コメント)

"The GIL makes it much easier to use OS threads"

Huh? How is this possibly true?

I agree with every point here apart from the concurrency one. A smorgasbord of async systems and os-thread-access-but-only-from-extensions is not a great concurrency approach in a multi-core world.

[–]the_hoser 15 ポイント16 ポイント  (11子コメント)

The GIL made it easier to use threads from the perspective of the CPython developers. Instead of maintaining the myriad of locks necessary to make Python keep chugging in the presence of OS threads, they kept it narrowed down to one lock. This allowed the developers of CPython to keep the design simple. It actually improves single-threaded performance, and single-threaded software was (and possibly remains) the most common use case at the time that the design decision was made.

Honestly, I think that the real issue is that the Python developers even attempted threading at all. Shared state threading in a dynamic programming language is simply a recipe for disaster. Message-passing parallelism wasn't really a popular idea in Python's early years. The design decision to mimic Posix threading haunts the CPython implementation to this day.

[–]kylotan 5 ポイント6 ポイント  (9子コメント)

Right, so what you're saying is that the original sentence I quoted is completely false from the point of view of an application developer. Which is fine.

However, to say it "actually improves single-threaded performance" is only true if compared relative to a hypothetical alternative that had some other, slower way of operating, and which incurred costs even when there was no second thread running (which would be unusual).

Honestly, I think that the real issue is that the Python developers even attempted threading at all. Shared state threading in a dynamic programming language is simply a recipe for disaster. Message-passing parallelism wasn't really a popular idea in Python's early years.

There are a lot of application domains where some degree of shared state across multiple threads is important. Games and simulations are two of them, for example. Whether those are suitable problems for Python or not is another matter, but some people don't like to admit these domains exist at all.

[–]masklinn 2 ポイント3 ポイント  (5子コメント)

Right, so what you're saying is that the original sentence I quoted is completely false from the point of view of an application developer. Which is fine.

It's completely true from an application developer's POV as well if that application developer does almost only IO and very little CPU-bound stuff.

[–]kylotan 0 ポイント1 ポイント  (4子コメント)

Why? Presumably you're talking about the situation where the GIL can be released for calls into external C libraries. But this itself is a workaround for Python's benefit, not for the extension developer. In a C/C++ app (for example), as soon as you make the so-called "blocking" I/O call that thread is suspended (pending a callback from the OS) and your other threads are immediately free to run on that processor with no further intervention from you. The difference with Python is that you have to explicitly tell it that this thread is going to leave the rest of the interpreter alone, so that it is capable of switching to the other threads!

[–]masklinn 2 ポイント3 ポイント  (3子コメント)

Why? Presumably you're talking about the situation where the GIL can be released for calls into external C libraries.

No, I'm talking about the inability to execute python in multiple threads at the same time leading to lowered chances of races due to the GIL, even if the developer is somewhat sloppy. Heavy IO means this isn't going to lower the overall throughput much so there's little drawback to it. Heavy CPU use means there's a high drawback instead.

[–]kylotan 0 ポイント1 ポイント  (2子コメント)

So, this restriction somehow makes it "easier to use OS threads" by (a) forcing people to write in a second language to access them effectively, and (b) limiting what they can do with them once they do that. That's not really a definition of "easier to use" I accept.

[–]masklinn 2 ポイント3 ポイント  (1子コメント)

You can't accept that something can be "easier to use" in a restricted number of use cases?

[–]kylotan 0 ポイント1 ポイント  (0子コメント)

No. They're not "easier". No code you write becomes shorter. It's not even clear that Python will prevent you wrongly accessing the interpreter from your C extension if you call the C API after having released the GIL. Nor does it prevent race conditions that arise from assuming values won't have changed while the GIL was released. I'm seeing zero benefit here and several burdens.

[–]the_hoser 2 ポイント3 ポイント  (2子コメント)

However, to say it "actually improves single-threaded performance" is only true if compared relative to a hypothetical alternative that had some other, slower way of operating, and which incurred costs even when there was no second thread running (which would be unusual).

There was an effort by a developer to remove the GIL and make the CPython interpreter thread-safe. The end result was a substantial decrease in single-threaded performance, even when no other threads were running. There was a long discussion about it on the Python-Dev list some time ago. I can't find it right now.

Even PyPy-STM is facing issues with single-threaded performance right now. However, I don't think it's fair to judge PyPy-STM right now, as it's still in heavy development.

There are a lot of application domains where some degree of shared state across multiple threads is important. Games and simulations are two of them, for example. Whether those are suitable problems for Python or not is another matter, but some people don't like to admit these domains exist at all.

I agree. I never said that shared-state multithreaded programming was to be avoided. I said that it's not really a good idea in a dynamic programming language.

[–]kylotan 1 ポイント2 ポイント  (1子コメント)

There was an effort by a developer to remove the GIL and make the CPython interpreter thread-safe. The end result was a substantial decrease in single-threaded performance, even when no other threads were running.

And that is arguably a flaw in the way Python works. It guarantees the integrity of the interpreter across threads, at a price. And that price doesn't even buy you guaranteed thread safety for your own code, sadly.

Other languages take a different approach, where the burden of safety is put more on the developer. If you want multiple threads, you have to coordinate access yourself, but usually each thread has its own interpreter that doesn't need to be shared and therefore requires no global lock.

How much does the 'dynamic' nature make it hard to isolate code from data and allocate one interpreter per thread, leaving all locking to the application developer? I don't know. It doesn't seem to be a problem for other VM-hosted languages, but does seem to be a problem for older languages that are designed for embedding and extending (eg. Javascript, Lua).

[–]the_hoser 0 ポイント1 ポイント  (0子コメント)

And that is arguably a flaw in the way Python works. It guarantees the integrity of the interpreter across threads, at a price. And that price doesn't even buy you guaranteed thread safety for your own code, sadly. Other languages take a different approach, where the burden of safety is put more on the developer. If you want multiple threads, you have to coordinate access yourself, but usually each thread has its own interpreter that doesn't need to be shared and therefore requires no global lock.

For sure. Many of these decisions revolve around keeping the implementation simple. There's something to be said for that, I suppose.

[–]selementar 0 ポイント1 ポイント  (0子コメント)

even attempted threading at all

Hey, worksforme. Not that I often prefer to use it, but nevertheless.

[–]stoned_economist 20 ポイント21 ポイント  (41子コメント)

As someone who has been learning Python the past 6 months, I've become very tired of introducing it to fellow coders. Almost immediately anyone who has heard of it before refers to it as a "scripting" or "web" language, as if it's just for kids who play around on the net or whatever. "Can you do graphics with it?" What? It is so disappointing to know more about C++ and Java than them, see and appreciate the differences, then watch them so quickly dismiss Python because their college professor or whoever told them it wasn't demanded by the industry. I expect less ignorance and less of a clique mentality from people who aspire to be "engineers".

I will be glad when Pythonic ideas finally dominate the industry and filter into other languages (like C++ 14) so the snobs can see what they've been missing.

[–]jinqsi 15 ポイント16 ポイント  (4子コメント)

What's strange is that my profs have been telling us that python is where it's at "in industry"

[–]stoned_economist 10 ポイント11 ポイント  (0子コメント)

Yes definitely recently.

[–]qudat 4 ポイント5 ポイント  (1子コメント)

Full time python developer here, can confirm!

[–]newpong 1 ポイント2 ポイント  (0子コメント)

Full time shitty python developer here, can double confirm!

[–]pythonautical[S] 17 ポイント18 ポイント  (22子コメント)

"Can you do graphics with it?"

The answer here is emphatically: "Yes, you can absolutely do 3D graphics programming in Python."

https://www.panda3d.org/

http://www.ogre3d.org/tikiwiki/PyOgre

http://www.pyglet.org/

Minecraft clone in 900 lines: https://github.com/fogleman/Minecraft/blob/master/main.py

[–]stoned_economist 9 ポイント10 ポイント  (4子コメント)

Yes obviously. My point was that I am always shocked when people have strong opinions about Python, yet haven't even looked into something as basic as what's been done with it.

[–]xiongchiamiov 6 ポイント7 ポイント  (3子コメント)

At least they're not complaining about how they'll "have to care about indentation". As if they'd ever get through code review without doing that anyways.

[–]oneonetwooneonetwo 4 ポイント5 ポイント  (2子コメント)

I've never really understood that whole category of complaint, really. I really do think less of someone who claims to be a programmer but gets caught up on such an introductory issue. You don't even care about indentation? Hey, maybe someone else should work on my business critical systems.

[–]p_j_w 1 ポイント2 ポイント  (1子コメント)

I'm an electrical engineer who frequently writes Python to do number crunching on somewhat large data sets. I'm a minimally competent Python programmer and yeah, indentation isn't a huge deal for me. The problem, however, comes when I share my code with other electrical engineers who aren't at all competent programmers but need to change some minor aspect of my code now and then. I can tell the people I instruct directly that 'you need to use 4 spaces instead of a Tab', but they're gonna forget that and if they hand my programs off to anyone else, they're certainly not going to tell that person. This leads to a lot of hassle.

Curly braces and semicolons never hurt anyone. They may not be as pretty as indentation and ending a statement with a carriage return, but they're no less readable. As far as I can tell, there's no good reason to not use them.

[–]oneonetwooneonetwo 0 ポイント1 ポイント  (0子コメント)

I don't see how not at all competent programmers are going to be any more comfortable with curly braces and semicolons, though. Whichever one you use, it's firmly in my suck it up if you want to tell the computer to do things pile. If you don't care about curly braces the compiler throws a tantrum, if you don't care about indentation the interpreter throws a tantrum. Pick your poison, really.

What I don't understand is the whole category of complaint, not which one is better.

Come to think of it, even in semi colon based languages, truly not caring about indentation at all means your work likely gets tossed at code review because it's probably unreadably inconsistent.

[–]kylotan 5 ポイント6 ポイント  (15子コメント)

The first 2 are wrappers around C++ libraries. The last one is a 2D library (and largely unmaintained).

Can you do 3D graphics using Python? Yes. Should you? Probably not.

[–]Kitanata 5 ポイント6 ポイント  (3子コメント)

There are pure OpenGL wrappers for Python. Here is the thing though, Mr. Hardcore graphics guy, if you're creating a game why recreate an engine unless you're building a high graphics AAA title and have millions of dollars and a team of hundreds of developers? Both Panda3D and Ogre are EXCELLENT game engines for most developers and have successfully been used in AAA titles. Torchlight for example is purely Ogre and CEGUI. If it's good enough for Runic Games, it's good enough for me.

Source: I'm a guy who's actually been involved in the development of high end graphics engines. Python is fine for 99.9% of game development work. (The .1% being the top dogs, of course.)

[–]kylotan 0 ポイント1 ポイント  (2子コメント)

Both Panda3D and Ogre are EXCELLENT game engines for most developers and have successfully been used in AAA titles.

Neither really match up well when compared against Unity or Unreal. That's little to do with the Python aspect though. It's just that if you're going to use someone else's engine, there's little point using one that's quite old and clunky just so that you can use your favourite language.

Personally I wouldn't try using a Python-based system on any game client where performance is an issue because you're going to need good access to multithreading and processing on multiple cores simultaneously, and Python makes that a real hassle.

[–]Kitanata 2 ポイント3 ポイント  (1子コメント)

A lot of that multithreading is handled in the C/C++ implementation though. Neither Panda3D or Ogre is implemented in Python. They just have python bindings which are "pythonic", and to my knowledge both engines utilize multithreading and multiprocessing fairly well. In the case you want to do multithreading or multiprocessing yourself, you can write that section in C/C++ and build python bindings around that implementation as well. It makes your code go from 100% C/C++ to 20% C/C++ (where performance really is an issue) 80% Python where it isn't.

[–]kylotan 0 ポイント1 ポイント  (0子コメント)

Of "that" multithreading? You make it sound like you'd never want to add any yourself. In my experience, adding background tasks that don't necessarily have anything to do with the rendering engine is very common.

In the case you want to do multithreading or multiprocessing yourself, you can write that section in C/C++ and build python bindings around that implementation as well.

Yeah, I don't call that "good access to multithreading". The game logic would normally all be in the same language. Switching languages for certain tasks, and being unable to call back into the main body of data effectively, is a serious roadblock in development.

[–]bit- 3 ポイント4 ポイント  (0子コメント)

Can you do 3D graphics using Python? Yes. Should you? Probably not.

I think this is the wrong question to ask. Right question is: Can you develop 3D software using Python? Yes. Should you? Absolutely!

It does not mean everything has to be python. It means you can write logic in python and development will be blazing fast. Core stuff that requires performance can still be written in lower level language and that is cool. Everything has it's place in this world. Panda3D did just that. All low level stuff in c++, high level goodies in python, game logic can be done in python. Everything actually can be done in python. See deferred rendering pipeline with global illumination written in python (for panda3d) here.

[–]elbiot 1 ポイント2 ポイント  (9子コメント)

Pyglet is 2D? No, pyglet is opengl. Google the game "ace of spades" to see a 3D game using pyglet.

[–]kylotan 4 ポイント5 ポイント  (8子コメント)

Pyglet is a 2D library using OpenGL. All it provides in terms of rendering capability is displaying 2D images as sprites, and some helper functions for low level vertex buffers etc. If you want useful 3D capability, you have to write your own OpenGL code for that and integrate it.

[–]elbiot 2 ポイント3 ポイント  (0子コメント)

I'll be darned. I never tried 3d in pyglet but I felt like the functionality was there and I was working around it.

[–]tjl73 1 ポイント2 ポイント  (5子コメント)

You can do 3D in Pyglet. There's an OpenGL example in the source distribution that shows it. Plus, I know we have a Pyglet backend in SymPy that does surface plots.

[–]kylotan 0 ポイント1 ポイント  (4子コメント)

Pyglet itself does not 'do' 3D in any meaningful sense of the term. The fact that you can use OpenGL with it makes it no different from using PyOpenGL with plain Python - you have to write all your own low-level code which is about 2 or 3 layers of abstraction below actually being productive.

[–]Cosmologicon 0 ポイント1 ポイント  (3子コメント)

Can you explain a little more what it means to "do" 3D? I've written a few 3D games in PyOpenGL and it didn't really feel to me like I had a bunch of abstraction layers in the middle. But maybe I just don't know what I'm missing.

[–]kylotan 0 ポイント1 ポイント  (2子コメント)

A typical 3D engine, or a comprehensive 3D library, will do things such as:

  • Load models and art info (including vertices, materials, textures, shaders)
  • Manage scene elements (movable models, fixed geometry, terrain, particles, lights, cameras)
  • Animate models (skeletal animation or morphing)
  • Decide what to render (based on a scene graph of some sort, maybe with occlusion algorithms, level of detail handling, sorting for transparency, etc)
  • Provide optimizations such as billboarding, batching, etc
  • Provide postprocessing via render-to-texture
  • other stuff

[–]Cosmologicon 0 ポイント1 ポイント  (1子コメント)

Okay, but a lot of that is important in 2D as well. I see what you're saying now, thanks, it just doesn't seem like a 2D/3D distinction to me.

[–]boarhogpython3 0 ポイント1 ポイント  (0子コメント)

Don't forget kivy.

My love goes to pyglet but there's only one person maintaining it and kivy is a damn nice alternative.

Hold strong, Pyglet!

[–]audionautics 4 ポイント5 ポイント  (1子コメント)

Eh, get used to it. It doesn't just apply to Python. With some people, anything other than the programming language they like is a cookie cutter one that you can't do anything with.

[–]stoned_economist 0 ポイント1 ポイント  (0子コメント)

True, I guess my statement represents some of the mentality I'm whining about.

[–]camel69 5 ポイント6 ポイント  (0子コメント)

You should tell them about the European Space Agency (and NASA as well, I'm sure) doing simulations and calculations and stuff for freakin' interplanetary spacecraft missions with Python.

[–]newpong 0 ポイント1 ポイント  (0子コメント)

are you related to drunk economist?

[–]haskell101 -1 ポイント0 ポイント  (0子コメント)

I will be glad when Pythonic ideas finally dominate the industry and filter into other languages

And what ideas would those be? Python got its best stuff from languages like Lisp that existed decades prior. I agree that it will be nice to see higher level ideas filter into the mainstream but to call them "Pythonic" is bizarre.

[–]jamesmcm 0 ポイント1 ポイント  (7子コメント)

Meh, people used to say the same about JS, but nowadays we have Node.js, Jquery, etc. and it's dominant.

Same with Django displacing Ruby On Rails.

C++ and Java are dead, web programming is the future as browsers replace OSes. Ignoring it is like sticking to C, Cobol or FORTRAN, 15 years ago.

[–]Kitanata 0 ポイント1 ポイント  (5子コメント)

As a guy who works heavily with both Django and Rails I have to say I don't see Django displacing Rails. They both have their place. That said, I prefer to use Flask. Haha.

[–]newpong 0 ポイント1 ポイント  (4子コメント)

I had a python meet up a couple weeks ago, specifically focusing on web frameworks, and i had never thought about before I said it, but there seems to be some truth:

Flask and the other light weight frameworks are designed for the developer. It is easy to develop and make work, but when you're building somehting for a client, you need django. It's more of a pain in the ass, but ultimately can do (just about) anything you need it to

[–]Kitanata 0 ポイント1 ポイント  (3子コメント)

My rule of thumb is, if you need it fast or its a prototype use django. If you're trying to build for scale use Flask.

[–]newpong 0 ポイント1 ポイント  (1子コメント)

weird. my philosophy is the exact opposite. But as I said in another post, scalability in web development is more affected by other infrastructure decisions than the code alone.

[–]Kitanata 0 ポイント1 ポイント  (0子コメント)

Flask makes it very easy to build horizontally scalable APIs.

[–]Mekhami 0 ポイント1 ポイント  (0子コメント)

that's a really shit rule of thumb.

[–]newpong 0 ポイント1 ポイント  (0子コメント)

who said django is displacing rails?

edit: as I re-read your entire comment, what the fuck are you talking about?

[–]Wolfspaw 13 ポイント14 ポイント  (1子コメント)

Great post!

I agree with those myth busters, and it's hard to beat the python ecosystem (batteries included indeed).

[–]guthran 0 ポイント1 ポイント  (0子コメント)

That's the best way to describe python. I'm stealing that.

[–]ShowMeTheMoons 3 ポイント4 ポイント  (7子コメント)

This diagram, linked in the article looks great.

Does anyone know what was used to produce it?

[–]boarhogpython3 1 ポイント2 ポイント  (2子コメント)

I would guess Python but I could be wrong

[–]ShowMeTheMoons 0 ポイント1 ポイント  (1子コメント)

Any Idea regarding potential libraries used?

I'm hinting http://graph-tool.skewed.de/.

[–]boarhogpython3 0 ポイント1 ポイント  (0子コメント)

No idea, honestly

[–]pythonautical[S] 1 ポイント2 ポイント  (1子コメント)

Haha nothing fancy, actually somebody did it on their phone: https://play.google.com/store/apps/details?id=com.drawexpress.lite&hl=en

I don't have enough experience to recommend it over anything else. The process of making a visually pleasing diagram is always slow and laborious. None of the diagrams were made specifically for the blog post, but were re-used from various internal presentations.

Edit: I will double check the exact program used.

[–]ShowMeTheMoons 0 ポイント1 ポイント  (0子コメント)

Thanks!

Nice article by the way!

[–]Kitanata 0 ポイント1 ポイント  (1子コメント)

Looks like LibreOffice Draw to me.

[–]ShowMeTheMoons 0 ポイント1 ポイント  (0子コメント)

I think you're right.

Thanks for the heads-up.

[–]thephotoman 5 ポイント6 ポイント  (5子コメント)

I'm currently writing a device emulator in Python, because it includes everything I need. Java has no built-in web server. C# doesn't distribute its web server. C...no.

So Python it is. Yes, I need another serial library, but that's true with every language.

[–]the_hoser 17 ポイント18 ポイント  (2子コメント)

I'm going to have to call you out on this one.

Java has no built-in web server.

The JDK definitely has a built-in http server framework:

package stupidhttpserver;

import com.sun.net.httpserver.HttpExchange;
import com.sun.net.httpserver.HttpHandler;
import com.sun.net.httpserver.HttpServer;
import java.io.IOException;
import java.io.OutputStream;
import java.net.InetSocketAddress;

public class StupidHTTPServer {

    public static void main(String[] args) throws IOException {
        HttpServer server = HttpServer.create(new InetSocketAddress(8000),0);
        server.createContext("/test", new MyHandler());
        server.setExecutor(null);
        server.start();
    }

    private static class MyHandler implements HttpHandler {

        @Override
        public void handle(HttpExchange he) throws IOException {
            String response = "This is my stupid response.";
            he.sendResponseHeaders(200, response.length());
            try (OutputStream os = he.getResponseBody()) {
                os.write(response.getBytes());
            }
        }
    }
}

I mean, it's not as simple to use as SimpleHTTPServer/http.server, but it does have one, and it gets the job done.

[–]thephotoman 1 ポイント2 ポイント  (1子コメント)

That's more work than import http.server; http.server.run_forever().

[–]the_hoser 11 ポイント12 ポイント  (0子コメント)

Oh, for sure. You won't get any arguments from me, there. I was merely pointing out to you that the statement "Java has no built-in web server" is false.

[–]ccampo 1 ポイント2 ポイント  (1子コメント)

Java has no built-in web server.

It doesn't need to. The Java servlet API allows you to built abstractions which you can run on any servlet container. Additionally, most servlet containers (tomcat, jetty, glassfish, jboss) have embedded versions which allow you to run a web server in a single JVM... and using these is usually as simple as including their library on your classpath.

Even that all being said, the JDK does have a built in web server since Java 1.6: com.sun.net.httpserver.HttpServer.

Also, C# Web API has OWIN, which is pretty similar to the Java servlet... again it gives you an API to create abstractions of web servers and allows you to self host outside of IIS. C# is definitely lacking here though, although the next version of ASP.NET promises to turn this around.

[–]haskell101 0 ポイント1 ポイント  (0子コメント)

This is good in an enterprise setting (which, to be fair, is what the topic is), but it's not very "batteries included".

[–]shadowmint 25 ポイント26 ポイント  (62子コメント)

Each runtime has its own performance characteristics, and none of them are slow per se.

Hahahahaha~

The more important point here is that it is a mistake to assign performance assessments to a programming languages. Always assess an application runtime, most preferably against a particular use case.

Fair enough, everything is relative, but this reads like a playbook for 'how to be defensive about how slow your favourite programming language is'.

What's with all the sugar coating? cpython is slow. Plugins and native code called from python are fast, and that results in an overall reasonable speed for python applications; but the actual python code that gets executed, is slow. There's a reason http://speed.pypy.org/ exists.

...but then again, pypy isn't really production ready, and neither are the other 'kind of compliant' runtimes like jython, etc.

It's pretty hard to argue with:

1) cpython is the deployment target for the majority of applications

2) cpython runs python code slow as balls.

3) overall, the cpython runtime is pretty much ok because of plugins and things like cython

4) python is a scripting language (wtf? of course it is. What is myth #4 even talking about?)

I mean... really? tldr; python is great for quickly building enterprise applications, but its strength is in the flexible awesome nature of the language itself; the runtime itself leaves a lot to be desired.

[–]zardeh 10 ポイント11 ポイント  (8子コメント)

4) python is a scripting language (wtf? of course it is. What is myth #4 even talking about?)

What does this even mean?

How does one define a scripting language? Like, the term doesn't mean anything.

Is bash a scripting language? Python? Any language with a REPL? Haskell? Lisp? Clojure? java?

[–]d4rch0n 2 ポイント3 ポイント  (0子コメント)

Yeah it's a bullshit attribute in my opinion. I don't know why you would call Python a scripting language. It depends on the fucking interpreter. If you're running python bytecode through CPython, it's a bytecode/VM language, but I don't believe anything in the Python language spec specifies that it has to run that way, or that you can't even compile it to some sort of machine code. Python isn't interpreted line by line by an interpreter, but that doesn't mean it's as fast as a compiled C program. Python is a programming language, the implementation is an interpreted/VM/compiled language implementation.

[–]Kitanata 6 ポイント7 ポイント  (0子コメント)

Anyone who uses the term "Scripting Language" and isn't talking about shell scripting pretty much loses all credibility IMHO. It is a derogatory term derived from ignorance.

[–]beltorak -2 ポイント-1 ポイント  (5子コメント)

The rule of thumb that I use is if you can take an arbitrary (but essentially complete) bit of functionality represented as a string in the language's natural syntax, eval it, and end up with something that integrates natively with the rest of the pre-written code, then it's a scripting language.

This is probably the least rigorous definition imaginable, but it does encompass many languages that are viewed as "scripting" languages, such as Python, JavaScript, PHP, and Ruby, but exclude traditionally-viewed "non-scripting" languages such as C and Java* . The fact that there is a separate pre-compile step to produce a "compiled" form (either as an intermediate "virtual machine" instruction set or an immediately executable hardware instruction set) doesn't enter into it at all. Any language implementable in such a way as to be run with an interpreter can (probably) be implemented with a pre-compile step, and vice versa.

But I'll admit that I do tend to fall into the lazy thinking habit of "scripting languages" as "not compiled".

* - DOS batch might violate this because it makes a difference if you run some commands directly with CMD /C ... or save them to a file. Fucking. Microsoft.

[–]kindall 2 ポイント3 ポイント  (2子コメント)

By that criterion, Lisp is a scripting language. (Well, the parse from string and eval are separate steps, IIRC, but...)

[–]beltorak 0 ポイント1 ポイント  (1子コメント)

Lisp is unique in a lot of ways. Wasn't it the first to implement a lot of concepts that are being rediscovered in modern languages?

[–]kindall 0 ポイント1 ポイント  (0子コメント)

Pretty much. Python's map, reduce, and filter are lifted directly from Lisp. It also has an apply function, although this has been deprecated since the introduction of the *args syntax.

[–]zardeh 2 ポイント3 ポイント  (1子コメント)

but exclude traditionally-viewed "non-scripting" languages such as C and Java* .

This ignoring the fact that I can implement my own "eval" function in java with reflection and in C with a bastardization of the command pattern, so I find this to be a weak definition. Just because a language comes prepackaged with a function doesn't define what it should be used for.

[–]beltorak 0 ポイント1 ポイント  (0子コメント)

but it wouldn't integrate natively with the rest of the pre-written code. For example, if I create a method and call it like MagicScript.createClass("public class Animal {}");, I cannot in the pre-written code do new Animal(). You would have to go through an entirely different process to make use of your new class.

[–]elbiot 16 ポイント17 ポイント  (1子コメント)

Agreed, having been focusing on performant code for a few years, I'd say python is slow. But, it has excellent wrappers for fast C code, and is easily extendable with cython or C when it really counts. I love python.

[–]Ikinoki 2 ポイント3 ポイント  (0子コメント)

there are compilers to c code available also. they improve speed a lot. nuitka for example

[–]d4rch0n 6 ポイント7 ポイント  (1子コメント)

Yeah. That's the only one I thoroughly disagree with. Python (CPython specifically) is slow, but it doesn't matter for the most part. People are writing shitty Java and Ruby and it doesn't matter if CPython takes a little bit longer to do something if it's written in 5% of the lines and 100 times more maintainable, so less fucked up bugs in the long run.

Of course, beautiful fast Java can be written that Python could never beat in performance, but for the most part performance IMO should also be measured in how long it takes to develop and squash bugs.

In a pure performance comparison, CPython can't match Java or true compiled-to-machine-code languages, but fuck it. Network speed is generally my bottleneck, not my sorting algorithm.

[–]tritlo 4 ポイント5 ポイント  (0子コメント)

Also, much of the functions in the stdlib is actually in C, so if you just heavily utilize those (like e.g. set or sort), you can get pretty performant python with very little hassle in my experience.

[–]istinspring 12 ポイント13 ポイント  (17子コメント)

Reddit is in Python, and this site it pretty huge. Language speed characteristics have relatively small impact. Nowdays there is more important things - What is more important it's architecture, 3rd party solutions, access to wide range of libs, ease of reading and writing code etc. For modern web apps it's just a wrapper between database and front-end.

And speaking about Python, the huge plus is ability to write asynchronous code, especially in python 3.

[–]Raphael_Miedl 6 ポイント7 ポイント  (14子コメント)

Sorry but gotta nitpick here. I wouldn't exactly choose reddit as an example here. It's a big site that uses python, but it is notorious for its "ouch we took to long" messages under peak.

That said there are certainly other big projects using python successfully with good performance.

[–]chub79 9 ポイント10 ポイント  (12子コメント)

Is it Python's fault here? Couldn't it be database, network, load-balancing, IO related?

[–]xiongchiamiov 7 ポイント8 ポイント  (9子コメント)

All of the above and more.

[–]oneonetwooneonetwo 0 ポイント1 ポイント  (8子コメント)

The suggestion even being that if the code was faster you could get more done on the same hardware.

[–]linuxwhisperer 1 ポイント2 ポイント  (2子コメント)

compared to developer time, hardware is cheap.

[–]oneonetwooneonetwo 5 ポイント6 ポイント  (1子コメント)

You've come in too early, we're not talking about developer time yet.

[–]linuxwhisperer 4 ポイント5 ポイント  (0子コメント)

oh, sorry. ill see myself out then.

[–]xiongchiamiov 0 ポイント1 ポイント  (4子コメント)

Not if I/O is the bottleneck.

[–]oneonetwooneonetwo -2 ポイント-1 ポイント  (3子コメント)

No, that's not really a case where you're doing more with the same hardware.

[–]xiongchiamiov 0 ポイント1 ポイント  (2子コメント)

Right; it's a case where you don't get more done with the same hardware, despite having faster code.

[–]oneonetwooneonetwo -3 ポイント-2 ポイント  (1子コメント)

Have I wronged you or something? We're angrily agreeing with each other.

[–]jimbobhickville 1 ポイント2 ポイント  (1子コメント)

Almost exclusively, I'm sure. I have yet to encounter a web or distributed system that wasn't bottlenecked on I/O.

[–]oneonetwooneonetwo 1 ポイント2 ポイント  (0子コメント)

In fact, I'm not sure what it would look like, really. Maybe something like an online zip file password cracker: one upload followed by intense computing followed by one download

[–]Kitanata -2 ポイント-1 ポイント  (0子コメント)

Those occasional issues aren't related to Python.

[–]surfingjesus 0 ポイント1 ポイント  (0子コメント)

also youtube

[–]newpong 0 ポイント1 ポイント  (0子コメント)

Most web apps aren't usually very computationally demanding, and there are plenty of other bottle necks(e.g., database structuring/connections/queries, caching, #/order of requests) that can be optimized to improve performance than the speed of the language alone, as such websites shouldn't be used for a measure of overall speed.

[–]justphysics 4 ポイント5 ポイント  (2子コメント)

oh goodness

that bottom plot on http://speed.pypy.org/

/r/dataisntalwaysbeautiful

(hope its not just me - in my browsers the xlabels are all on top of eathother)

[–]superdaniel 1 ポイント2 ポイント  (1子コメント)

I get the same thing with Firefox :(

[–]oneonetwooneonetwo 0 ポイント1 ポイント  (0子コメント)

Same with chrome.

[–]billsil 3 ポイント4 ポイント  (12子コメント)

2) cpython runs python code slow as balls.

Unless it's written under the hood in C. There is no reason for mathematical code to be slow in Python. There is no reason for parsing code to be much slower than C especially since the standard formats are coded in C and are available in Python.

[–]d4rch0n 5 ポイント6 ポイント  (11子コメント)

Yeah, but at some point you're coding in C, not Python. If you write every high performance part in C and call it through Python, how much can you really say it's Python?

Don't get me wrong. That's probably the best way to do high performance stuff with Python, but I don't think it means CPython is fast, it just means it uses a fast C API.

[–]billsil 4 ポイント5 ポイント  (6子コメント)

If you want to. I use numpy, so while I have to vectorize my code and call the right functions in often non-obvious ways, it's still technically pure python.

Somebody did coded it in C, but that doesn't mean you have to.

but I don't think it means CPython is fast, it just means it uses a fast C API.

CPython is running the code, so I say it counts. If all the standard library was written in Python instead of C, everyone would say Python is slow. Instead, they say it's fast enough. That stuff counts.

[–]yen223 1 ポイント2 ポイント  (2子コメント)

Numpy isn't pure python, is it?

[–]billsil 3 ポイント4 ポイント  (1子コメント)

No. A fair amount is written in C, but some is also written in Fortran. My understanding is most of scipy is actually written in Fortran and is just a wrapper around LAPACK.

[–]tavert 1 ポイント2 ポイント  (0子コメント)

most of scipy [...] is just a wrapper around LAPACK

For dense linear algebra, yes. There's a lot of functionality in SciPy aside from dense linear algebra though. Some of the underlying libraries are Fortran, some are C, some features are custom C++. According to https://github.com/scipy/scipy the breakdown is 38.3% Python, 25.8% Fortran, 18.6% C, 17.1% C++.

[–]tritlo 1 ポイント2 ポイント  (0子コメント)

The key here is that I'm still writing pure python, but I'm utilizing someone elses C code. If you argue that's not enough python, then every use of linpack in other language should be disbarred.

[–]d4rch0n 0 ポイント1 ポイント  (1子コメント)

I still draw the line when you're bringing in machine code into the Python process memory and it's not running bytecode loaded from pyc files. It's fast, but it's actual CPU instructions, not Python bytecode first.

Of course it counts. Again, I'm not saying it's terrible, and that it shouldn't happen, or that it's a flaw. I'm just saying the fast parts aren't Python and I wish that the interpreter/VM implementation was fast enough so that we wouldn't need to use C code to have high performance programs. Any programming language could interface with C/fortran libraries and be high performance. It doesn't mean that that language's interpreter is fast though.

I would like to see an implementation that uses purely the Python language and still be high performance.

[–]tavert 0 ポイント1 ポイント  (0子コメント)

I would like to see an implementation that uses purely the Python language and still be high performance.

You already have that with PyPy. Unless you don't mind C extensions not working, what most people want in practice is a fast implementation that would be C-API compatible with CPython and extensions. Unfortunately that's extremely difficult as the C API is pretty closely tied to the slow internals of CPython.

I suspect users aren't really all that picky about implementation language, but something easier to read and contribute to would be nice for maintainers' sake.

[–]fnord123 1 ポイント2 ポイント  (3子コメント)

That's fine and correct. But I think it misses the point: we discuss language performance characteristics is so we can get an idea of the expected performance of an implementation and assess the risk of being limited by our choices. If you choose CPython then your limitations are mitigated since you have one of the easiest paths to hook into a C implementation of the workhorse part of your code. Also jumping across the FFI is pretty quick in Python.

[–]d4rch0n 0 ポイント1 ポイント  (2子コメント)

Sure, Python applied through CPython and C libs will be fine. This is the way I suggest doing things if performance is required and the initial Python implementation is too slow (but always first Python unless we KNOW it's going to be slow).

Generally network speed is my bottleneck for almost everything I do, so I can just use gevent and get perfectly fine performance.

Still, I don't think performance regarding this is the problem to solve. The hardest problem to solve here is having good C programmers, and all of which goes with that, like memory, freeing pointers and nulling them, code security, etc. If your high performance part hasn't been done by a third party, you need to rely on your skillset in your team and this stuff isn't trivial at all.

That means higher skilled devs, which means higher salaries, and also a lot more development time. You lose a lot of the applied benefits of Python, like super-fast development and being able to pull in anyone who is decent with Python and not having to worry about use-after-frees, etc.

Python is definitely my favorite language and the one I'm best at, but it's a serious consideration that I feel limited if I rely on having to fall back to C if I need high performance. I love C, I'm just not very confident, and I'll have to really take time to ensure code safety and correctness.

Even if I'm just using pre-built C libraries, I still need to worry that I'm using them 100% correctly and not opening up a security issue due to the way they're supposed to be used, or even that the original developers wrote safe code.

[–]fnord123 0 ポイント1 ポイント  (1子コメント)

You don't need to write it in C. You can use Cython and get like 80% of the speedup[1]. I mean, your Python program begins its life at potato speed as though you were using Perl or even Ruby. If something isn't performing well enough you move the inner loops (almost) verbatim to a pyx file and jiggy your setup.py and then you get something at about Java performance (or potato quality C code - fast, but not hand crafted shit off a shovel speeds). Then if it's still not fast enough you can get these supposed elite developers to crank out some C to squeeze out even more performance.

There are a lot of options to get results based on the amount of work you put in. In a business environment this is sweet since you can time box a lot of the improvements and make actual progress with each sprint.

[1] Bullshit made up number. Take it with a grain of salt.

[–]d4rch0n 0 ポイント1 ポイント  (0子コメント)

That's some cool stuff. I haven't seen that before.

There is definitely some learning curve to writing Cython code, but it's still a very neat trick without having to code raw C. I see your point.

I still wish we had a faster reference interpreter than CPython though.

[–]kenfar 2 ポイント3 ポイント  (0子コメント)

You apparently have already decided that python is a "slow as balls" scripting language.

However - "scripting language" is not a well-defined term, and is often in a context like this meant as a derogatory description: the local java team arguing that project x shouldn't be done in python because "it's only a scripting language".

And fast or slow are so relative that to describe a language like Python as slow is also meaningless: does this mean every application written in it will be slow? does this mean you can't process trillions of transactions in it? does this mean it's merely a toy?

While I would like some Python operations to be faster than they are today, I have processed a hundreds of billions of complex transactions using cpython - and performance wasn't on my top 4 list of challenges.

[–]lambdaqdjango n' shit 3 ポイント4 ポイント  (4子コメント)

OK, the Python interpreter is slow, but in most Web project Python is light years faster than tomcat + J2EE shit in all develop, setup and serving speed.

Yeah, some of your fancy for loop Java programs may be faster, but I have yet to seen one myself in production. Especially those enterprise SSH java ones.

Anyway, that's my own observation. YMMV

[–]the_hoser 5 ポイント6 ポイント  (2子コメント)

The research disagrees with you.

[–]lambdaqdjango n' shit 2 ポイント3 ポイント  (1子コメント)

How about write speed?

http://www.techempower.com/benchmarks/#section=data-r9&hw=peak&test=update

There are tons of tricks to optimize for read/write speed, for example you can check source code for Python vs Java in the "Single Query" round. All java has fancy MySQL Prepared Statements in ORM level with connection pools, yet many of the php/python ones are constructing new SQL text and connection for each HTTP request. That's why it's slow.

[–]the_hoser 1 ポイント2 ポイント  (0子コメント)

So write a better benchmark and submit it to them. They have a well laid-out contribution process on their GitHub account. You seem to know how to optimize web applications, so they could benefit from your experience in representing various frameworks.

[–]istinspring 0 ポイント1 ポイント  (0子コメント)

I got same arguments from Java programmer i know... Oh you don't even have static typing, that could lead to problems! Oh you don't have this and that.

But then i aked, dude are you code something which required light fast speed and such large applications that static types is so critical for you.

And also i saw a website he made (very slow and really outdated), hell i can do same in few hours in python. With less code, more easy to debug, using wide range of awesome frameworks.

[–]jimminy 2 ポイント3 ポイント  (2子コメント)

What kind of examples could you give of Python's slowness causing great problems in real-life applications?

Raw execution speed doesn't matter much any more actually (like in the 90's). If it did, everyone would just use C or assembler. Practically all software is I/O bound anyway so database queries are the real bottleneck. For tasks requiring raw speed there are ofcourse the possibility to use C routines from Python so even that is not a problem.

What matters instead is the speed and ease of development and you just can't beat Python in it.

[–]shadowmint 5 ポイント6 ポイント  (0子コメント)

Any maths.

Not everything is I/O bound; specifically for data processing (eg. splunk) and scientific computing, python uses c heavily, because it's just too slow to be remotely usable otherwise.

[–]kylotan 1 ポイント2 ポイント  (0子コメント)

What kind of examples could you give of Python's slowness causing great problems in real-life applications?

Raw execution speed doesn't matter much any more actually (like in the 90's).

Simply not true. There are various areas where speed is still important - video games, simulation, scientific data crunching, artificial intelligence and machine learning.

In some of these cases, Python turns out to be fast enough. In other cases, it does not.

The last performance problem I had with Python was implementing planning/pathfinding algorithms. Python's requirement to allocate everything on the heap via pointers meant that exploring a large search space was very expensive, in terms of allocation costs and cache misses. That could have been mitigated if I could have offloaded it into a background thread, but Python's poor at that too.

[–]Veedrac 1 ポイント2 ポイント  (5子コメント)

...but then again, pypy isn't really production ready

Where do you get that idea?

[–]shadowmint 0 ポイント1 ポイント  (4子コメント)

Where do you get that idea?

http://pypy.org/compat.html

[–]elsjaako 3 ポイント4 ポイント  (0子コメント)

That's just saying that the C API isn't production ready, and not all libraries are supported. If you target your development at Pypy it's production ready.

[–]Veedrac 1 ポイント2 ポイント  (2子コメント)

That's like saying Clang isn't production ready because it doesn't support all GCC extensions. PyPy is extremely compatible against the Python language.

[–]shadowmint -1 ポイント0 ポイント  (1子コメント)

...but we're not talking about the python language we're talking about python as a viable target for enterprise applications, which means tangibly using 3rd party libraries, that will almost certainly have c plugins.

[–]Veedrac 0 ポイント1 ポイント  (0子コメント)

That's true if you're trying to support already-built Python code, but if you're building something new that's rarely a problem because for most use-cases there's a PyPy compatible port or equivalent.

[–]kindall 1 ポイント2 ポイント  (0子コメント)

This reminds me a lot of Paul Graham's famous article on how Lisp was the secret weapon of Viaweb. http://www.paulgraham.com/avg.html

I have no particular difficulty believing Python is PayPal's secret weapon. I have done a fair bit of scripting in Python, and have found that Python code does what I expect the very first time, more so than any language I've ever used. Compared to Python, coding in JavaScript or (shudder) VBScript is like walking uphill through 6-foot deep snow both ways.

Except if Python really is PayPal's secret weapon, they shouldn't have written this article, 'cause now it's not so secret anymore.

[–]westurner 2 ポイント3 ポイント  (0子コメント)

Tooling, strong conventions, and code review are what make big projects a manageable reality.

Thanks!

[–]novagenesis 0 ポイント1 ポイント  (0子コメント)

I'm surprised to hear so many people agree that their friends think Python is a toy language. It feels like the whole "new, hip, useless, scripting" sticker has more than worn off on Python in my world. I've never been face-to-face with someone who thinks Python fails to stand as an enterprise language.

For what it's worth, it's also a top-choice language for Machine Learning and for Predictive Analytics. Heck, I took a class in statistical modelling that chose python over R.

[–]Paddy3118 0 ポイント1 ポイント  (0子コメント)

A scripting language is also a controlling language. It is that language embedded in larger applications that makes those tools subject to your will Embed them in an office suite or a CAD program and suddenly you can do the repetitive, effortlessly. Give then access to libraries, and deep science, cloud computing, or GPU's become accessible.

There is greatness in scripting languages!

[–]laststance -1 ポイント0 ポイント  (9子コメント)

I think the biggest with Python is that you need to learn to forms of syntax. Or at least keep in mind what you're doing between Python 2 and Python 3. Since in a work environment, you might be asked to maintain a program that was written in two, while developing new code in 3.

Sure you could rewrite the code written in 2 and bring over to 3, but management has to sign off on that decision.

[–]Sinistersnarefrom knowledge import * as karma 7 ポイント8 ポイント  (7子コメント)

Python2 and 3 are not drastically different that you need to use two forms of syntax. Please give me a concrete example of a difference in Python 2 and 3 in which there is no compatible solution between the two.

The rumours that there is a difference is a myth, a few extra import statements to have it work in both 2 and 3 will not kill anyone, and management will not have to sign off on anything.

[–]Rubykuby 2 ポイント3 ポイント  (0子コメント)

Yeah. I've always find this argument kind of odd. I just write code that works in both versions. Of course it'll bork up at some lines, but an extra import or extra if statement easily fixes that.

Just write valid Python 3 code. If it ceases working for 2.7, fix it.

[–]d4rch0n 3 ポイント4 ポイント  (5子コメント)

Here's how to convert 2 to 3 in about 99% of Python programs.

s/print (.*)/print(\1)/g

[–]oneonetwooneonetwo 1 ポイント2 ポイント  (1子コメント)

The 2to3 program is really handy as well. I think if print statements weren't changed then the switchover would have been much more popular. I think a lot of people got scared when even Hello World had to be changed.

[–]Sinistersnarefrom knowledge import * as karma 0 ポイント1 ポイント  (0子コメント)

The problem with 2to3 IIRC is that it was only for creating python programs, not multi version compatible code bases. At least that was originally the problem.

[–]Ikinoki 0 ポイント1 ポイント  (2子コメント)

ehh. how about bytes and unicode

[–]Sinistersnarefrom knowledge import * as karma 1 ポイント2 ポイント  (1子コメント)

Most code that people write are not applicable to that problem. Mostly it is library code that worries about that, and when they make the switch they should be able to handle both.

[–]billsil 0 ポイント1 ポイント  (0子コメント)

Someone still has to tell the library what the encoding is. I run a library that tries to support unicode. Everyone wants it to just know the encoding. Sorry, I can't do that. Text editors like Notepad++ can't do it either.

[–]erewok 1 ポイント2 ポイント  (0子コメント)

I work in a shop that has both Python2 and Python3 code running among a variety of projects and it's pretty dang easy to go back and forth between the two environments.

[–]jeannaimard -5 ポイント-4 ポイント  (0子コメント)

2 years ago, I made handsome bux with python, to convert data from a 15 year old system to transfer to a 35 year old system (yes, 15 to 35).

It worked, even though the client was utterly retarded (as you may guess).