From techmeet
Links
Intro / Indy.gr Setup
qwerty: hello Zapata
qwerty: anybody else for the plone/indycore meeting?
qwerty: occam, PseudoPunk ?
qwerty: The development site is again available at http://dev.indy.gr
qwerty: you can try publishing articles and creating projects
Zapata: hi qwerty...
Zapata: anyway...
Zapata: let's wait 10 more minutes
qwerty: sure
Zapata: and let's then have a short meeting
qwerty: agreed
Zapata: I haven't been able to do a lot, due to my job situation, but coming week I'll have a lot of time
Zapata: I'd like to work on some stuff and understand the set up better and its implications for a possible large scale indymedia implementation
Zapata: I didn't manage to complete my installation btw...
Zapata: but I'll retry it with the latest versions and if I fail again, I'll ask you ;-)
qwerty: ok. it shortould work now
Zapata: so
Zapata: no sign of anyone else yet
qwerty: no. and I couldn't find Clopy either
Zapata: I know that occam did some work...
Zapata: I'm sure he'd like to talk about it...
Zapata: anyway
Zapata: let's just do it with the two of us now...
Zapata: like...
Zapata: you have been working on open publishing?
qwerty: yes. it's not complete but you can try it
qwerty: I also fixed a big problem of the previous version
Zapata: which problem?
qwerty: now there are no article objects created when going to the publish form, untill you succesfully submit your article
Zapata: right
qwerty: no disk writes at all
qwerty: but there are still pieces missing on the interface
qwerty: like publishing photos
qwerty: more info
qwerty: and comments
Zapata: ok
Zapata: so, what are you going to work on next?
qwerty: the infrastructure to support it is there but we have to blend it with the new UI
qwerty: I'll be improving the open publishing, adding special views for audio video and photo objects
qwerty: using the plone4artists products
Zapata: ok
qwerty: which are already in the bundle
Zapata: my personal plan is the following:
Zapata: 1. get it installed... if I fail by myself, I'll ask you
Zapata: 2. find some piece of small functionality to work at so I can get a good feeling on how it works. I'll ask you for input on what this small functionality could be
Zapata: 3. I'm going to write some sort of draft review, which can be the basis of the report we can make as a collective
qwerty: great
Zapata: and I'd like to have all of this done during the coming week
qwerty: you could start with making changes to the existing templates which are at indycore/browser
qwerty: and then learn how to define new browser views
qwerty: and work on UI screens that are not there yet. like the Forums, Images, Audio, Video sections
Zapata: ok
Zapata: we'll have a better discussion on this once I have installed it properly
qwerty: you can find the templates and the view classes of the new UI at indycore/browser
qwerty: whatever is under the skins directory applies to the old UI (no zope3 views)
qwerty: I guess I should document that
Zapata: right
qwerty: so. do you want to discuss any specific implications for a possible large scale indymedia implementation as you said?
Zapata: well
Zapata: let's wait for that
Zapata: especially occam has some thoughts about that
Zapata: it would be better to have this discussion with him...
qwerty: ok
Zapata: I had dinner with occam about a week ago...
Zapata: and we talked some things over about plone
Zapata: he has talked to some dutch techies with a lot of experience
Zapata: perhaps, the best thing to do is...
Zapata: hang around a bit in this channel
Zapata: and wait for occam to show up or so
qwerty: ok. I'm looking over his report once again
Zapata: ok
qwerty: about the ZODB, I definately agree that it has it's disadvantages. However there are big ZODB deployments (I've seen a few) and the people at the last Plone conference were optimistic that the time of the ZODB will come
qwerty: I met andycat from engagemedia in that conference and we talked about Plumi and indycore
qwerty: about the memory requirements, it's true that you need at least 2GB for production with squid enabled. 4GB is even better
qwerty: but if we could combine 2 or 3 of those servers in a load balanced cluster, they may be able to host several high traffic sites
qwerty: the zope page templates are also quite slow compared to other template engines but there is a new template system with the same syntax that it's under development that outperforms all the others according to Alex Limi (one of the founders of Plone)
qwerty: and finally there is the repoze project which seems to be an important development for the zope/plone world
qwerty: http://repoze.org/index.html
qwerty: at the front page there is an entry on setting up repoze with deliverance is a very interest component
qwerty: it's part of the opencore full stack and it allows easy theming
qwerty: opencore will be probably moving towards repoze and Plone3 soon, so indycore will go along
occam: oi
occam: sorry, i had to devliver some stuff...
qwerty: hello occam
Zapata: occam!!!!!
Zapata: occam!!!!!
Zapata: occam!!!!!
Zapata: occam!!!!!
occam: yes
occam: gimme sec do read the backlog
occam: ok
occam: http://www.techmeet.org/txt/Plone
occam: did you kids read the "summary"?
qwerty: yes
occam: my more realistic thoughts about using plone at the moment are kinda... sceptic... but i think some things are possible..
occam: first, we had to use the CFMDeployment and make some kind of a Mir Copy with Plone
occam: CMF
occam: even with caches and a few loadbalancer, we just don't have the resources in the network to run such a plone site
occam: because of the 2G memory problem...
Database / ZODB
occam: also, i really would like to try the sqlalchmie thing
occam: paulsa said we would miss some ZODB features... like change-trakc thing.. but we dont really need this
occam: it would be much better if we could use existing databases
occam: to avoid the migration problems, and to avoid allot more setup work...
qwerty: a practical solution for this that I've seen in a few large deployments is to move old content to an sql database
occam: i mean, we would need to setup a ZODB on another host to have a half-way performant setup...
occam: with sql we could just use a already existing mysql/psql setup..
qwerty: I've seen many zeo setups and the zodb server never seemed to consume many resources. the bottleneck is usually on the zope side
occam: how much memory does the zddb server take?
occam: usaly the db boxes have around a 1g memory that is completly used by the db...
qwerty: it depends on the cache you let it have
occam: but you can adjust these abit
occam: k
qwerty: [[1]]
occam: In our case, with about 700,000 objects in the ZODB, memory consumption is around 1GB on a cluster of 3 ZEO clients (with 4 threads and 30,000 objects in cache each one).
occam: germany.indy is at article id 201773
qwerty: well we can test it for ourselves. I could add a million articles on the development indycore site and see how it performs without any caching enabled
occam: dev.indy.gr is a seperated box?? can we have a login, or can you install collecd to we have some nice graphs?
occam: collectd
qwerty: it hosts a few other sites as well
occam: uhm, hard to tell the performance then ;(
qwerty: can anybody else provide a clean testbox then?
occam: my desktop here is the only 2g mem box is know ;)))
occam: i know
qwerty: we could try with a 1gb box and see if it does the job
occam: it would be interesting to know if we can scale it down, yeah
occam: if there is some way to disable modules
qwerty: I never tried it on 1GB but I have several zope servers in a 2GB box
qwerty: occam: which modules?
occam: i dunno, the standard plone i have running here seems to be huge and overloaded with features...
occam: i would love if i could just disable allot of them
occam: like the zodb reversioning thin
occam: or the admin interface
occam: ;)
qwerty: I'm not sure if they consume memory just by being there if you don't actually use them
occam: yeah.. dunno, if its good code it doesnt.. but if it gets included everywhere or so..
occam: also, paulsa said it it nos such a good idea to mix things like opencore and plone4artists...
qwerty: why not?
occam: there would be conflicts, memory usage would get hight etc..
occam: it sounds a bit strange for me too to just plugin opencore, plumi and plone4artists and use some of there features
qwerty: actually I'm trying to integrate the low level video and audio functionality from p4a into the opencore/indycore interface. I think I can make it work
occam: i dont really have any deatils or inside knowledge ;(
qwerty: ok
bonzai: hi - sorry for the delay
occam: btw, why did you used opencore?!
qwerty: for the functionality it provides on creating projects, people profiles etc
qwerty: I think it would be great to allow people that visit an indymedia site to get into action by trying to form groups, communicate their goals, etc
qwerty: and because opecore has a big, talented and well funded team that develops it
qwerty: http://dev.indy.gr/newswire/test-news-article/09_son_de_la_barricada-1.mp3/view
qwerty: on the above link you can check out the plone4artists audio interface and functionality on the old indycore skin
qwerty: I haven't tested it on production yet but so far the p4a components dont seem to introduce any problems in the mix. There were problems with Plumi however
qwerty: at least with an old version of Plumi. things might be better now
bonzai: well plumi is based on p4a with special fixes for plumi that is the reason why u have troubles . at this moment it wil lalso not work smoth with the newer plumi stuff
qwerty: bonzai: are you working on Plumi?
bonzai: also but mostly with plain plone and or p4a
bonzai: another reason why i can be really tricky to mix opebplan and the rest if the upgrade issue
bonzai: ah
bonzai: openplan/opencore
- paulapoly has joined #cms
bonzai: hoi paulapoly
paulapoly: hi bonzai
paulapoly: sorry i'm late, had urgent stuff to take care of
qwerty: hello paulapoly. are you also working with Plone?
paulapoly: yep, that's sort of what i do for a living
qwerty: me too
qwerty: have you checked out indycore? there's a development site at http://dev.indy.gr
qwerty: there is also http://www.techmeet.org/txt/Plone
paulapoly: no, haven't heard about it. Looks like plone, though
qwerty: it's based on opencore
qwerty: the software behind openplans.org
qwerty: opencore is based on plone but includes other external components
occam: hi paulapoly, hi bonzai nice to have you here
qwerty: indycore is a layer on top of it that enables indymedia like functionality. see http://trac.indy.gr for more
occam talked with paulapoly and bonzai about the indy setup... based on that i made the summary
bonzai: qwerty: btw we do not met at the conference in naples, or ?
paulapoly: qwerty, you were in Napoli?
qwerty: Yes. My name is Dimitris.
qwerty: I met andycat from engagemedia there
paulapoly: ah, the physicist from greece?
qwerty: no, that was george. a friend and plone partner
paulapoly: sorry, i'm bad at remembering names...
paulapoly: you were at our funny little NGO-meeting in the side alley?
qwerty: yes
qwerty: I was standing most of the time
qwerty: there weren't many chairs
paulapoly: ah, i slowly remember. I'm the one from Friends of the Earth Netherlands
bonzai: small world :)
qwerty: I left some stickers of indy.gr, out website
qwerty: indeed! I remember you
occam: cool
occam: paulapoly: did you had time to look at the summary?
Requirements / Server Setup / Traffic
paulapoly: on the imc requirements: i would still say the 'affordable hardware' requirements should not mean you cannot use Plone. Yes, i would recommend 2 gigs or 4 gigs of ram, but that's about 50 euros these days
paulapoly: hell, i have about 32 gigs of ram just lying around, to give away to projects
Zapata: 50 euroes is a fortune in some places
paulapoly: the requirements are kind of mixed-up anyway. There is no way you can use "affordable hardware" (if that means an aging pentium2 machine) and "video/audio" at the same time, I'm afraid. Video needs loads of bandwidth and storage, unless you just store references to YouTube
Zapata: that's very true
Zapata: anyway, perhaps there should be some sort of survey of imc servers on their specs
Zapata: and put that next to plone's requirements
Zapata: compare it to it I mean
paulapoly: It's also a question of organisation. A benefit in western europe should raise a couple of thousand euros, easily.
bonzai: yup i agree
occam: paulapoly: its also about flexebility, so you can move a application from one server to another in a short time.. thing is, we have all kinds of server in the network, spread all over the world usaly, with 2-3 techs resposible for them, sometimes more.. then we just have vservers donated by a tech collective etc pp... even if you have physical access and it is your own box and you have the money for some more memory and the board is able to take more... you still need to go over, and put it in, with a usaly busy tech, this can take a week etc...
occam: having a lightweight system is kinda very importand ;( from my experinces of the last years
occam: and i do indy stuff since 2001
paulapoly: i see the point, but i feel the definition of 'lightweight' has changed since 2001. And i do not only operate in Europe as well.
paulapoly: set up some sites for Myanmar opposition this year, they had pretty impressive hardware (512 mb vserver, more than enough for a smaller plone site with mainly text/images)
occam: so, lets say you move the zodb to a different box, you could setup a indy plone that would only take 512mem?
occam: i give you some stats...
occam: sec
paulapoly: well, that depends on stuff like number of visits, traffic,
qwerty: occam: not if it has a lot of users that can log in and have personalized pages
qwerty: but otherwise you probably can
occam: paulapoly: http://getajob.ath.cx/stats/ that was g8 in germany
occam: note: Multiply all numbers (file sizes, hit numbers) by 1.5!
occam: i think we had 8 static mirrors and 2 media file mirrors
occam: but that was mostly to spread the bandwidth
occam: not because of the cpu
occam: ;) all shtml SSI
paulapoly: i'm not really impressed. Should run on a vserver with 512 megs
paulapoly: well, not if all of those people log in and have personalised pages
paulapoly: but then again, what's the percentage of people actually loggin in to indymedia? <5% would be my guess, but that's just a guess
occam: if we enable login at all
bonzai: why you want to do that ?
Zapata: there are some reasons
Zapata: to reserve and protect a nick
Zapata: to deliver personalized content
Zapata: to allow for social networking
occam: 6327063 Hits a Day. << your sure?! i mean, i know some php apps that get in trouble with that amount of traffic
qwerty: occam: without logins plone performance is only constrained by the rate that squid can serve pages
paulapoly: "hits" is a very inaccurate measure, very dependant upon the cms as well. "Visits" is a lot more useful
occam: paulapoly: we dont log IP's so visist its useless
Zapata: but an estimate can be made
Zapata: at indy.nl, it's 6-10 page views per visit
paulapoly: but again, I'd say that indymedia Germany should be able to give itself decent hardware. If not, that's a definite organisational problem. I'm sure the stats for indymedia azerbaidjan look a lot friendlier
paulapoly: there are some fun ideas as well, we could run our own OpenID provider. That way, you could have your own nick on all the indymedia sites
paulapoly: a bit scary as well, i admit, but fun nevertheless
toya: the thing is that if people start to help down in south america by using the connections we can get on universities i believe most of the machines will have around 1 giga memory
occam: why? we are able to make such a huge event with small hardware in a very flexible way.. our only problem is.. we dont have enough developers for our cms...
toya: or even 512
toya: the hardware wont be fancy but the connection is good and is free u know so that should be counted
toya: (just adding some thoughts in the midle of the conversationg :P)
occam: the lack of developers is the main reason why we started the cms discussion... and we kinda where hopeing that there is something as flexible as Mir
paulapoly: Mir is hardly lightweight, now is it...
Zapata: mir is quite light weight actually
paulapoly: java?
occam: well, the static producer is very lightwight.. also, my main idea for plone at the moment is, to make a Mir like setup with the CMFDeployment..
Zapata: why would the mere fact that it's written in java make it heavyweight?
paulapoly: because it needs a server (tomcat, or whatever you're using) that will always be heavier than thttpd
Zapata: erm...
Zapata: why?
paulapoly: because java (and zope, btw) have things like a security model, an object model and other stuff to deal with
Zapata: sure
Zapata: but still
Zapata: it entirely depends on the kind of application you run in it
occam: PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
occam: 31695 tomcat55 18 0 859m 482m 3588 S 0.0 47.2 1173:08 java
Zapata: a mir installation only does the publishing and generation part...
paulapoly: duh, that doesn't say a thing. The FOEI Netherlands server is visited quite a lot (about 80 gigs traffic a month) and is usually at 0% when it's just serving up cached content
occam: but keep in mind, only admins and ppl who use the search or the publish funtion actualy ever see the real Mir
paulapoly: but it's a different thing if there are 20.000 people logged in at the same time, producing content
Zapata: the thing is that mir has a light weight strategy
Zapata: you can write any kind of software on any kind of platform and produce a light weight and a heavy weight system
Zapata: the thing about java in particular that add overhead is it's virtualization aspect and possibly it's memory model
Zapata: but then, is there any technology that is suitable for web application development that doesn't have similar things?
Zapata: not that I would say we should java for anything, but I'm just kind of bored with the focus on the underlying technology
paulapoly: yep, agreed
Zapata: the underlying technology is a small factor in making something work
Zapata: in the indymedia case, I believe organization is the major factor
paulapoly: i'm just saying that zope *could* do the job, but that it will take careful caching
paulapoly: the big advantage is that for instance using buildout, you can fully automate the setup of NGINX plus varnish.
paulapoly: That means, a setup is easy to repeat and deploy
Caching / Static Sites
occam: can you go more in the details about caching? we are all techs here, we will understand ;)
Zapata: one important aspect is, that none of the sizeable imcs work the same way
paulapoly: well, the idea is of course to serve up the content that is available for anonymous by not going to the cms at all
Zapata: which means all of them will need to be set up differently
Zapata: and this will reflect on what knowledge an imc needs for its caching needs
paulapoly: the fun part of the varnish/zope combination is that zope can actually contact the cache by itself, to say that some content is new or updated.
Zapata: that's a very nice feature
paulapoly: so the cache never has to invalidate itself, it just gets told what to do
occam: so by caching you only talk about the varnish stuff, not about plones internal caching features?
bonzai: no and yes :)
paulapoly: internal caching is dependant on machine configuration
paulapoly: like memory availabe
paulapoly: so that will change from machine to machine
paulapoly: it's kind of silly to set a 512 mb cache on a 256 mb machine...
bonzai: but the fun thing at least for me is the combination from varnish and plone internal
occam: so, i already learnd that there is a zodb cache.. lets say whe move the zodb to another box and let it use the mem for cache so its not swapping, what more can we dont on the zope/plone side?
occam: what is this template compiler thing?
occam: pre-compiled templates
qwerty: occam: there are many levels of caching. You can easily cache the newswire instead of generating it each time a request reaches the zope server. But in a social networking app there will be a lot of stuff that cannot be cached
paulapoly: that's internal zope magic. But pre-compiled templates don't make all that much difference, although there's constant work in the zope community on it
occam: so every object you have on the side, you can say "cache this"?!
paulapoly: you can cache each object, but the difficult part is permissions
paulapoly: even if it's cached, you will still have to check if this particular visitor is allowed to see this object
paulapoly: and it becomes more complex if the object is for instance a news feed, for which the current visitor is only allowed to see 34 out of 57 objects
qwerty: indeed
occam: can you disable the permissions system?
paulapoly: so that's why in general you can only achieve super-good caching on content that is available for anonymous
paulapoly: anything else will always be restricted by the security mechanism. That is only solvable by throwing more hardware at the problem, be it clusters or whatever
occam: thing is, i am happy if i have the same shit running as it is at the moment..
occam: ;)
paulapoly: the same shit should not be a big problem, in my opinion
qwerty: I think there is s a question of strategy that we have to answer. Do we want Indymedia 2.0 to be personalized and allow social/activist networking or do we just want to become more resilient from server crises by minimizing the hardware requirements as much as possible?
occam: then we can think about adding more features like login or whatever
occam: so, by putting objects into the anonymous level, you can half-way get around the permission system?
paulapoly: yep
ryan: hiiiiiiiii
ryan: i want to say something!
occam: hello ryan :)
paulapoly is away for 10 minutes, have to run to shop to improve the tobacco situation...
occam: kk, i have some food here, lets make a little break
qwerty: I have to go soon as well. But ryan please speak up first!
ryan: i cant right now i am being harrassed by my mom
qwerty: ok. I'll probably be back later
Decentralisation / Cluster
occam: qwerty: to ansfer you question, for me its not about web2.0 or server cirsis, but the server crises has priority and i don't wanne increase the crises by putting up the requirements we have for a box
paulapoly: the stress (and 'heavy' requirements) would be on the ZEO box (or cluster). But you can very easily add ZEO clients (with caching) that only connect read-only to the database. They are very fast, and only need very modest hardware. So just to serve up the content to anonymous, you can quickly add more machines
paulapoly: you can do that real-time
paulapoly: and, with buildout-recipes, it takes no hardcore skills at all.
paulapoly: that means, you can quickly add more machines at G8 summits, or whatever. You just need to add the new servers to a round-robin DNS pool, or put pound in front of it
paulapoly: only drawback is that for publishing, you'd have to go to another URL (like publish.inymedia.blah), but that is more or less what mir does now too, iirc
paulapoly: the ZEO cluster at the centre also has more options for failover than the current setup. It does need some planning, though. Luckily the guru of those setups is Wichy, former Debian project leader and a nice guy. A few belgian beers as bribe, and i'm sure he'll help out
paulapoly: in any case, it seems i'm just rambling on here all alone, but i'll give some tips about performance and plone from my experience
paulapoly: * don't mix plone and php-apps on the same machine. Problem is, for php you need the apache2-worker model since php doesn't do threads properly. That kills performance
bonzai: no no i am still here :)
paulapoly: * if you just run plone, you can use the very efficient NGINX webserver in front. Or apache, but then the new apache2 models which work much better
occam: would work ZEO over long distances?
occam: over SSL?
paulapoly: ZEO works over long distances, although not over SSL. But that's why tunnels were invented
occam: how hard is the ZEO setup?!
occam: just a buildout?
paulapoly: yep, it's a buildout. The tricky part is the DNS round-robin, or use Pound as load-balancer
occam: we do allot of RR with mir..
paulapoly: but that's a bind problem, not a zope problem
occam: and yes, publish is usaly seperated
occam: sperated box in most cases
paulapoly: the ZEO setup can get complicated if you want the ZEO servers to use the same disk-storage. You need a network-aware filesystem then
occam: hmmm
paulapoly: so most setups have a ZEO cluster in the same physical location, sharing a disk storage
paulapoly: and then the ZEO clients can be spread around long-distance
paulapoly: the public only sees the ZEO clients anyway
occam: zeo clients != zeo cluster?!
paulapoly: no, the ZEO cluster is the database, and just the database
paulapoly: the clients are Zope instances that connect to it, and do all the publishing
occam: ok, but we could do a zeo cluster just for replication over long distance via tunnel..
paulapoly: to make it perhaps clearer, our setup @work is the following: it just runs one ZEO server, and then 3 ZEO clients. Two for anonymous publishing, one for logging in
occam: the zope client <-: zeo communication.. what kind of protokoll is that?!
paulapoly: all of these run on the same physical machine, by the way
paulapoly: it's just to make better use of today's multicore machines
occam: i see
occam: is there a diff between a ZODB Cluster and a ZEO Cluster?
paulapoly: nope, same thing
occam: ok
paulapoly: but a ZEO cluster over long distances is probably not a good idea
paulapoly: it could lead to long write delans
paulapoly: delays
occam: only for replication, to have the data save on another place
paulapoly: it's then better to have another ZEO cluster at a remote location, and replicate the database
Deployment
occam: yeah
occam: ok
occam: i have this thing i tried to explain to you before
occam: lets say
occam: we have plone setup A and plone setup B
occam: A gets search.site.indy and B gets admin.site.indy...
occam: and then we have C,D,E whatever, that would just get site.indy and be in a round robin
occam: all connect to the same ZODb
occam: BUT, i would like to have this via some kind of a "build" out..
occam: you see, if a user surfs around on C,D,E it would click on the search and go to search.site...
occam: but the zope setup is all the same, only the url is different
occam: it would be no problem at all if zope is using relative urls
occam: so we can set some fixed urls at the search form, etc...
paulapoly: i don't quite get why you would want to separate the search
occam: just performance...
occam: "module" "decentralisation"...
paulapoly: the search is (in the end) done at the ZODB
paulapoly: i would prefer it if A, C, D and E can all search
paulapoly: more decentral
occam: hmm... well
occam: thing is..
paulapoly: searching is quite efficient anyway. It's only done on indexes
occam: Zapata said this before, we usaly have 3-10 imcs runnin on one box
occam: there is this long-term idea of having a global search index of all imc's
occam: same for tags
occam: also
occam: well
occam: forget the last part
occam: thing is
paulapoly: ouch, that opens up a whole can of worms. Searching in chinese and arab can be tricky
occam: i would like to have the modules work indepenet... that you can shutdown the admin of the zodb, and the search would still work...
occam: or the..
occam: so, i would run a replicaed zodb on the search box in this example
occam: but, for example, i could setup the search of 4-5 imc on the same box
occam: maybe because its a faster box with more memory
paulapoly: that would work, although the multilingual stuff takes some care
occam: is there some "read-only" flag?! that you can really be sure that a plone setup would never change anything in a zodb?
bonzai: what u mean ?
paulapoly: yep, just connect to the zodb as read-only ZEO client
bonzai: ah ok, yes
occam: i see
occam: well, you did not ansfer my question about the url
occam: ;)
occam: search.site would be the same zope as site.indy.. just a different url and a bit different cfg
occam: is that possible via some build-out?!?!!
occam: i mean, no big diff if i just go and make a svn co of the current setup
occam: and do the cfg chnages by hand..
bonzai: if u use nginx you can do it via the buildout afaik, paulapoly ?
occam: buildout = deployment right?
paulapoly: i don't quite get the problem still. All Plone url's are always relative to $portal_url,
occam: ok, can i change $portal_url as part of the build out... say, i have buildout A for search.site with $portal_url=search.site... and then i have site B.. etc...
paulapoly: buildout is a bit more than that. It's kind of a recipe that does deployment, indeed. In end effect, it creates a kind of isolated environment, so that you can be sure you have your own correct version of python, zope, etcetera
paulapoly: so you're not dependant on the system packages of the machine you're working on
occam: its a complete python env?
paulapoly: occam, you don't even need to specify the URL in the buildout. It doesn't need to know. It will happily use whatever Apache or NGINX say
occam: ok, no wonder you don't get the idea ;))))
paulapoly: occam, yes. Well, you can use the standard python env, but sometimes that's a bit tricky if you deploy it on different versions of linux
bonzai: plone wants to have 2.5
paulapoly: if you compile your own, the buildout recipe will even work on Windows (ok, bad idea, but still cool that it actually works)
bonzai: ah
bonzai: 2.4
occam: hmm, i really wanne get a "skelton" small plone running..
paulapoly: use buildout!
bonzai: ok wait :)
bonzai: here comes the url:
occam: are there any more things you can disable?
occam: ;)))
bonzai: https://selene.waag.org/trac/radar/wiki/DownLoad
paulapoly: it's cool. The first time it will download the internet and compile the world, but it's cool to watch anyway
bonzai: its with install manual
paulapoly: bonzai, what's that url?
bonzai: the radar test buildout :)
bonzai: normal plone 3
bonzai: only with another theme so far
paulapoly: ah, ok
occam: well, i have a plone running on the mac here.. question is, how do i get it small :)
bonzai: i hpw i will be able to releae the 'real' radar buildout in 2 weeks
bonzai: which kind of install do u have ?
occam: 3.0.2
paulapoly: define "small". And anyway, how does someone who owns a mac blabber on about 'cheap hardware' anyway :-) (Sorry, couldn't resist)
bonzai: lol
occam: hehehehe
occam: it was the payout of my holliday
occam: after i quit the fkn job
paulapoly: occam, check the plone site anyway for buildout. There were some slight issues with it, or at least with Leopard. You need to ensure you have a proper gcc
bonzai: i still have to buy a new notebook ....
bonzai: hehe
bonzai: my buildouts runs also on leopard
bonzai: but u are usuing tiger anyway or
occam: sill on tiger
occam: happy
paulapoly: it's just a matter of ensuring you have gcc, as far as i know. Then it should work
paulapoly: oh, and subversion
bonzai: check my manual or plone.org :)
occam: i used this mac dmg something
bonzai: its with extra osx manual
bonzai: https://selene.waag.org/trac/radar/wiki/MacUserPage
bonzai: and you want to have mac ports
occam: i dont, i make most of the stuff from source
paulapoly: ok, then you probably have everything already
occam: ;)
bonzai: u need
bonzai: python24
paulapoly: the best buildout i've seen so far is the one for Primagis
bonzai: libxml2
bonzai: py-pi
paulapoly: that's a whopping combination of Plone, Mapserver, and tons of other GIS software.
occam: oh, 3.0.3 is just out i see
bonzai: py-pil
bonzai: fucking keyboard ...
paulapoly: it really downloads the universe and then compiles it. Took about 7 hours on my humble 1 ghz 512 mb laptop
paulapoly: but comes complete with tweaked geospatial version of postgres
paulapoly: and can then do utterly cool stuff with geodata
bonzai: hehe i want to test it , sometime :)
occam: gosh
occam: did i said skelton?
paulapoly: skeleton = buildout
occam: ok, but, i already have a plone running, how do you make a build-out of my current non existing site setup?
paulapoly: you don't, or at least not completely automatic. You can make a buildout that includes your current add-on products,
paulapoly: and then just copy the Data.fs over
paulapoly: then everything should be fine and dandy
paulapoly: not much work, really
paulapoly: all the important info from your current site is in Data.fs
occam: so the buildout is only for the python files.. not the config
occam: well, kinda.. i think i get the idea
bonzai: which config ?
occam: like, i said object xy is cached... ab is not...
bonzai: well
bonzai: what i do
bonzai: i setup a buildout
bonzai: configure such things
bonzai: and than i have them in the buildout, also for the next time i will setup these buildout
bonzai: and i copy
bonzai: the data.fs
paulapoly: occam, about the caching: you usually have to set that yourself. You do that in what is called a 'policy product'. Basically, it's a Plone product that changes a vanilla Plone site into your site. The settings in there are just XML files, and you can easily generate them from a 'live' plone site
paulapoly: so, you tweak a site until you like it, then dump all those settings into XML files, and create a minimal product whose only job it is to apply these settings to a fresh site. So, in a way, it's part of the buildout
paulapoly: but it's more clean to separate it into an "indymedia.policy" product. That means it's easier maintained separately in a subversion repository, or whatever
bonzai: yes
paulapoly: the buildout then just contains instructions to download the indymedia.policy product as part of a new setup
occam: i see
paulapoly: separation is good. You can then have different people working on the "indymedia.policy" and "indymedia.fancylayout" products without getting in each others way
occam: is plone4articis also just a "product"?
bonzai: nope
paulapoly: actually, it's a bundle of about 5 products
bonzai: right
paulapoly: they've separated it into audio, video, calendarstuff, and some infrastructural products
paulapoly: because audio needs things like ID3 tags, which don't make sense for calendaring. You can install the separate products if you just want audio but no video
occam: http://dev.indy.gr/newswire/test-news-article/09_son_de_la_barricada-1.mp3/view
occam: qwerty just did that
paulapoly: yup, looks like plone4artists.audio
occam: puhh
occam: where the hell is Zapata
occam: Zapata
paulapoly: and, actually, "Plumi" is just plone4artists with some extra skins and stuff thrown in. It was a separate branch of plone4audio, mainly dealing with transcoding using the indytube code. But it most likely will be merged into one again.
occam: quit your fkn poker game
Zapata: yo
Zapata: what's up
occam: did you followed?
occam: do you have any more questions??
Zapata: some parts I followed
occam: ideas?
qwerty: for p4a video check this one: http://dev.indy.gr/newswire/barsandtone.flv/view
occam: for the new world order...
occam: qwerty! :)
occam: a true test video
paulapoly: nice
Zapata: indeed
Zapata: anyway
Zapata: the information provided is very handy indeed
qwerty: you can also publish links to external videos (e.g. from youtube) and p4a.video will collect the metadata and embed it in the site
paulapoly: i do also like the fact that plone4artists has very good integration with YouTube and the like. Of course these are evil corporate entities, but if we really want to provide video capacity to indymedias with very limited resources, i think we can only realise this by using their resources. No way that indymedia zimbabwe can host their own videos
Static Site Producer
occam: any updates about the CMFDeployment?!
bonzai: ok i have to go now - sorry
bonzai: tot later
occam: thanks bonzai
occam: cya
paulapoly: no, but there also interesting other developments there. Nothing quite definite yet, i just happened to be in the bar with Wichy lately
paulapoly: i'll keep you informed
paulapoly: but even a handmade wget+rsync solution works quite well to get static deployment
paulapoly: but you don't get the fancy search options
occam: cool..i kinda prefer CMF/static compared with varnish/cachs.. because a static site work independet from the backend
occam: i know, but that wget things ia horrible hack ;)
occam: also
occam: in Mir, we dont generate the full sire..
paulapoly: yeah, CMFdeployment is more elegant. Still, you still have the search issue. Of course you can index & search static sites with external tools, but it doesn't come close
occam: what search issue has CMF?!
paulapoly: well, it generates static html. So no live-search with fancy ajax widgets
occam: (in Mir are able to generate parts of the html and use SSL for the includes etc, this we have have seperated the Newswire and other stuff.. so we don't need to generate the hole site each time)
occam: ahh.. ok
occam: cant you redirect the ajax shit to another box?!
paulapoly: yes, but that kind of defeats the purpose of backend-independance and single point of failure, does it?
occam: i mean, thats the idea here.. every "dynamic" stuff would get redirected to another subdomain
occam: well.. it would still display the content, if the fancy ajax search is working or not...
paulapoly: I'm still kind of convinced that the perfomance issues can be worked out without resorting to static html.
occam: if our mir server is down, we usaly just edit the index.shtml with vim and tell the ppl that publishing/search is done for a while ;)
paulapoly: but of course it's still an option as 'last resort' or extra backup
occam: well, thing is, it fkn scales like shit.. you can get hit by NYT, CNN and whatever, and the only thing you have to worry about is the bandwidth
paulapoly: a read-only ZEO client, coupled with a proper varnish/NGINX setup scales about the same as plain HTML, is my experience
paulapoly: in reality, you serve up everything from cache anyway, which makes it equivalent with static html
occam: yeah, but compare the resources it needs..
occam: yeah, i know, and again, the cache depends on the backend..
paulapoly: a vserver with 256 or preferably 512 megs should do
paulapoly: nope, cache is just plain disk-cache
occam: well, if your luck you have everything in the disk cache?!
occam: the hole site?!
paulapoly: yep. Only issue is of course video.
paulapoly: same goes for plain html, by the way. You do need space to put the whole site as plain html.
occam: 1.5G de.indymedia.org/
occam: most of 700mb or so it is the "icon" stuff, image thumbnails
occam: we have seperated the images on a media server
paulapoly: otherwise your current mir setup also wouldn't work, would it? I mean, if it holds just part of the site as plain html, you can't survive long when the mirserver is down
paulapoly: 1.5 gigs? hmmm, my usb key holds more, i'm not that impressed :-)
occam: yeah, thats what i mean ;) you can tar.gz that and you have another mirror in 15mins..
occam: beat that :)
paulapoly: so, yes, i would cache everything on disk with varnish. Should be easy. And easily rsynced onto another machine
occam: yeah, but even you say cache everything, it would not be a 1:1 image of the site, because you dont know if somebody read the article from 2002/12 whatever
paulapoly: then again, i have nothing against holding an extra static copy in plain html. But i just think they're equivalent, in terms of performance
occam: yeah..
occam: guess so too
paulapoly: occam, yes it is. Rembember, Plone tells varnish when an article becomes available, independant of wether someone requests it or not
occam: so, the first time you start plone+varnish, it would tell varnish to cache everyhing?
paulapoly: yes you can. That will take a while, obviously.
occam: thats wsgi isnt it?!
paulapoly: then again, no worse than Google spidering your site
occam: so, plone knows what varnish has in the cache?
occam: even if the first "cache all" fails, it would not have to start from scratch?
paulapoly: no, it doesn't. It just knows to tell varnish whenever something's added or updated
paulapoly: Varnish has to do the initial spidering when plone starts up. But it does that anyway, when for instance Google comes along, which is about once a day on the Friends of the Earth website
paulapoly: but after that, varnish will faithfully serve up 'old' content when it can't reach the plone site
occam: puhh
occam: now i just need to learn plone..
occam: ;)
paulapoly: it's fun, although i admit it can be a steep learning curve. But believe me, when you've seen some of the code, you don't want to go back to crappy Joomla/Drupal :-)
paulapoly: probably Mir is quite smart as well.
occam: is there some nice graph about the strcutire of plone? that usaly helps me allot of understand
occam: ah
occam: one last thing
occam: i forgot you ansfer about this one
paulapoly: nah, not really. I could do a session somewhere the next weeks, if you're still around
occam: local/shared code
occam: easy... i will just rtfm and try around
paulapoly: we're (that's Sisi, Bonzai and me) are trying to get Plone Bootcamp organised in Amsterdam somewhere in the spring
occam: we have this thing with mir and sfa, drupal also has this feature, that you can use one base and run different local version/sites withit
paulapoly: that's 5 days of training, by the ultimate plone teacher, Joel Burton
paulapoly: we can probably get very good rates (or even free) for some of you
occam: ai ai.. good reason to come over to ams
occam: hehe, guess zap and me would be interested ;)
Multi Site Setup
paulapoly: occam, i don't quite get the local/shared thingie... explain a bit more
occam: well, you have shared codebase, the core of the application/cms
paulapoly: the training would be at the Waag
occam: this codebase gets reused all the time..
occam: then you have the local site setup
occam: with your modifications up to a certain level...
occam: templates and shit
occam: mir even has this "plugin" system where it allows overwrites / extenions of funktions
occam: but it would still use the one core..
occam: so, if you have a bug in the core, you fix the core and its fixed on all the sites using that core
paulapoly: that's kind of 'automatic' if you use buildouts in plone
paulapoly: the more important question is, do these local sites use the same data?
occam: no
occam: they have there own database
paulapoly: there's also PloneMultiSite, which does that. FOEI international uses that
occam: and cfg etc..
paulapoly: there, you publish an article at the 'admin' site.
occam: lemmy check
paulapoly: and then say it should be published to the SouthAmerican site, the European site but not the Asian site.
occam: a MultiSite Setup for plone would be very interesting
paulapoly: all of which have their own templates and stuff
occam: http://plone.org/products/plone-multisite for the record
occam: aehm
occam: nothing on there ;))
paulapoly: but beware, it quickly becomes very complex if you have to deal with X sites and Y translations. Sort of a classical X*Y problem
occam: yeah, we know..
occam: version problems etc..
paulapoly: should the spanish translation go onto the asian site, or only the english one? That kind of stuff
occam: well, its not so much about sharing articles, more about using the resources a better way
occam: having multiple imc sites on one box
paulapoly: the resources thing is solved quite well in plone. The language thing still remains a headache
occam: if you have a bounch of small sites...
paulapoly: the buildout framework is a relief, i must say. You can be flexible about versions in it, it's easy to say: "I need the Forum product, version =: 2.12 but not bigger than 2.17"
paulapoly: but still, this whole local code thing sounds like it's really not an issue under plone
occam: hmm
Admin Interface
occam: are there some alternaitve admin interfaces?!
occam: we really do allot of batch operation things..
occam: i only say this build-in fancy in-the-side editing
paulapoly: normally, what you do is write python scripts to do that, and cron them
occam: scripts can't make the complex moderator descions...
paulapoly: but there is also this concept of "Content Rules", which is quite brilliant
occam: its like having 200 comments in batch, reading them at one on a big page while makring a "hide this shit" checkbox
occam: lord, sorry
occam: its like having 200 comments in a batch, reading them at onec on a big page while marking a "hide this shit" checkbox
occam: you should see the de.indy admin and have some fun with nazi comments/spammer
paulapoly: yes, you can. Although the complex decision would be to include 117 of those 200 comments, so you would still have to read them, won't you?
paulapoly: but of course you can simplify the webinterface for power users
occam: yeah, the question was, do you know any power-user admin interface?!
paulapoly: VIM
occam: hehehehe
paulapoly: you can do amazing stuff with the "External Editor" extension. It sort of mounts your entire site as a file-system
occam: true
paulapoly: I have my site mounted as file-system, and use standard linux tools to copy/move stuff around. I edit articles in Nano.
paulapoly: (yes i know, but i hate vim. I like nano)
occam: its not about edit, its about changing a flag
occam: hidden or not..
occam: but nice to know
occam: its this webdav thing right?
paulapoly: you can change the publishing state using Nautilus
occam: or do you do a chmod 0000 to hide it?
paulapoly: probably konqueror as well
paulapoly: but i'm a gnome person
occam: i see
occam: i guess i know some imc kids who might use this feature... but most will stick with the web
paulapoly: web interfaces are always kinda sucky, but i must admit the new ajax-like stuff in plone3 makes it easier for the point'n'click crowd
paulapoly: at least Plone can easily provide comfort features like RSS feeds for "comments not yet moderated" and stuff, which makes it easier
occam: hehe, because of the amount we never really thought it about a rss
paulapoly: yeah, it's probably a bit much. Although a Firefox Extension for moderating would be so cool....
paulapoly: there's lots of discussion in the plone-world now about this. Although the first focus was more for generic sites, not necessarily indymedia stuff. So the first focus was on comment-spam-prevention, captchas and automatic spam removal
paulapoly: but i'm sure i could kick up a discussion about easy tools for moderators. I suck at javascript, and at firefox XUL programming, but there are smart people out there
occam: hmmm
occam: i gave up on XUL, i could not fill a fkn list box ;)
paulapoly: yeah i know, i tried also. You have to slaughter a small furry animal at full moon to make it do anything
occam: XUL would be nice, but we also need some very easy to use interfaces, and by easy, it also means that the interface should be simple so it could work a on very old p2 box or so
occam: no ajax or so..
occam: for these ppl who use the inet for the first time...
paulapoly: well, web-interfaces will always be kind of clunky unless you use ajax.
paulapoly: so if you have lots of comments that need to be removed, that will mean lots of clicks
paulapoly: the only thing i can think of, that it's quite easy in plone to define keyboard shortcuts. So you could define a function key that means "kill this comment, and jump onto the next comment"
occam: cool
occam: i never understood why ppl dont provide more keyboard controlls on websites
occam: like cursor controll for slideshows
paulapoly: plone actually has about 15 or so in the standard config
paulapoly: problem is, you don't actually see their definition on-screen unless you use Lynx (or a braille reader)
paulapoly: so most people never get to know them
paulapoly: because who reads help-files anyway :-)
occam: ok, guess ppl just have to catch up and setup things and play around
occam: but was very good to get all these infos together
occam: so, big thx again ;)))
paulapoly: ok, i'm off. contact me again
