Siggraph 2014: Day 4

AUG 13 2014, WEDNESDAY

WEDNESDAY 0900..1000

  • user case SVA AT NYC
  • open pipeline
  • priority
    • render speed
    • job frame count
    • thesis prgoress
    • submission time
    • limit by the users
  • improvement
    • using shutgon
    • monitoring student progress
  • crunch time
    • tracke
  • pipelinefx -> cube
    • they are using shotgun and cube
  • future impromvemnt
    • utilizae asset managmentment
      • seamless progcessing
    • better integration with shotgun
    • work and render over chould services
    • limit concurrent jobs per student
    • assign individual render days for seniors
    • show eta all the time
    • how many threads per job
    • hyperthreading
  • render management software
  • discussion points
    • automation vs learning the manual way
    • universal pipeline management system for education
  • windows
    • use AD to control user directory
    • qube -> tag the job with your name
    • qubeproxy user -> service
  • how to define # of threads
  • ACTION ITEM
    • way to scale the job
    • check out “open pipeline”
    • check out the thread per job and hyperthreading

WEDNESDAY 1000..1100

  • openscenegraph
  • state of play : robert osfield
  • all the stuff will be on the OSG website
  • osg-3.2.1 stable release
  • JMVP joint medical visualization project
  • digital learning foundation -> go check the website
  • volume rendering
  • scripting
  • osg ui
    • using qt5
  • improvements
    • script support
    • create custom widgets in lua
    • create behaviours/callbacks in lua
    • modifying scene graphi within lua
    • complete user interface in lua
  • osg-3.4 release plan

WEDNESDAY 1030..1130

  • study on the cloud rendering
  • guy from a montreal
    • chris -> director of technology digital domain, 20 yeras,
    • coal peterson -> business render
    • -> studio cloud
    • dave psoar -> softimage, autodesk, mentalray, ILM Rnd in light and rendering. in montreal
    • daren gran -> consultant, CTO DD, CTO dreamworks animation
  • dashboard and monitoring
    • render farm -> squeze every dollar out
    • becomes like a noise function
    • visualization of the realtime data is really important
  • IO, cpu, swap, thread
  • loving islon
  • next gen animaiton session ->
  • creative iteration -> and fit them in budget
  • human engineering
  • cloud
    • manager in the cloud
    • all thing in the cloud?
    • SAS ->
    • overflow render workers
    • amazon DWS
    • performance issue?
    • configuration nightmare
    • studio cloud
    • join AWS submit
      • C-section
      • new industry
  • user interface of the future render farm
    • web based is way to go
    • open up
    • concerned about the security
      • they can’t see the name of the jobs from the other showss
    • RESTapi
    • control apis, browsers
    • next gen ui customization -> web based
  • commercial product
    • are you happy customer?
    • reporting, artists friendlyness,
  • rendering 2D
    • error reporting
    • look up the log or python traceback
    • different ui for different departments
  • scheduling interesting in animal logic
    • bulk of the jobs are rendering but processing
  • singular of inhouse schedular
  • optimization
    • utilization? throwput?
  • compute scheduling, render scheduling
  • segarate all of your data
    • connect the file
  • gamifying the submission
    • web view ->
    • public shaming
  • sprung and elastic search
  • pixar -> john, susan fong,
  • digital domain ->
  • pipeline -> event driven
    • rabbit mq
    • using event backend and render management
  • hooking up the scheduling based
  • properl data coming in
    • stats dat
    • graphis
  • gpu computing
    • really great at gi computing -> beauty pass
    • depends on the the projects
    • evolve so fast
    • problem with the driver
  • nvidia open c??
    • dedicate the
  • very diverse
    • dollar intensive
    • low cost commotiy
  • ACTION ITEM
    • get the info about next gen animation sessionx
    • join AWS submitt ->
    • add security in the crazyq.. per projects, login based

WEDNESDAY 1130..0130

  • exhibitions

WEDNESDAY 0130..0230

  • cg in usa + canada
    *
  • full sail university, winter park, florida

WEDNESDAY 0100..0200

  • vray: method stuido
    • transforer movie
  • fx
    • on volumentric elements simulated in houdini and rendered in vray

WEDNESDAY 0200..0300

  • vray talk
    • ILM, Jan, captain america
  • using vray for katana
  • vray – 3 years, gi-joe
  • using render man for a while
  • started doing alembic
  • ACTION ITEM
    • check out open image io
    • https://sites.google.com/site/openimageio/home

WEDNESDAY 0300..0400

  • vray talk
    • scannline, 300
    • 25 years in cg, 5 yrs in
  • max and maya pipeline
  • vray is a core, used since v1
  • flowline => major scanline tool -> tightly integrated with vray
  • scannline
  • goes to custom vray shader
  • 8K texrtures
    • ??? exr
  • change some of the animations
  • vray shader out of box works well
  • flowline -> watersolver, fluid solver,
    • fire, smoke and water
    • anything takes less than 12 hours in the farm
  • questions
    • reusing fx sim -> can’t do
    • incamera depth of field ->
    • vray-RT
  • ACTION ITEM
    • check out flowline
    • ask hubbard about this?
    • ask hubbard about the best texture format for vray

WEDNESDAY 0400..0500

  • vray talk
    • kevin marko, chaos group
    • kevin, sup blur 12 years
  • blur reel 2013 lorez
    • league of legends
    • bioshock infinity
    • load of the rings
    • batman and justice league
    • starwars
    • halo
    • starwars
  • batman archam night
    • game trailer
    • WB -> rocksteady was a client
    • they have game models in progress
    • hair -> vray hair system
    • hair shader
    • love architectures -> can save the geo
    • looped the building 800,000 lights
    • more shader test
    • all animation done by the mocap
  • thor 2 prologue
    • thrusday morning
    • concept design
      • helmet, shield
    • got digital doubles from vfx
    • upraise them
    • used vray skin texture to create skin variation and differnt morph targets
    • making hugo
    • creating a cloud
      • card changing to the different camera view
    • showing different angle of the image sequences based on the camera
    • battlefield look dev
    • nuke and mari pipeline
  • construct
    • nvidia
    • entirly gpu rendering project
    • vray -rt gpu
    • 500 sec -> 45 sec renders
  • GPU rendering
    • small team
    • massive computational power
    • small footprint
    • make it more responsive
    • vray rt -> only on max?
  • rendering final
    • renderd in 5 days
      • 3x 3dboxx 4920 ->
      • 3x nvidia k6000’s
      • 6x nvidia k40s
      • 3dboxx 4920 gpu edition
    • construct teaser at 4K, stereo, 60fps
    • used nvidia vca cluster
      • reduced a 10 year workstations cpu -> 5 days
      • 14 min on single vca
      • took 9.5 hours on vray cpu
    • VCA ??
    • vray for motion builder
      • virtual camera capture
      • ability to capture a virtual camera location
      • compose in real time to photorealistic scenes
    • realtime rendering is happening in motion builder
    • http://www.boxxtech.com/products/workstations/single-socket/3dboxx-4920-gpu
    • roughly around $20,000
  • how did you get a fund?
    • sponsers -> boxx, nvidia, chaos group, autodesk motion builder,

WEDNESDAY 0500..0700

  • opengl es 3.1
  • 11 years of opengles
    • 2002 working group
    • 2003 1.0
    • 2004 1.1
    • 2007 2.0
    • 2012 3.0
    • 2013 3.1
  • what’s new since sig 2013
    • release 3.1 at GDC
  • es 3.1 goals
    • bringing ogl 4.x to mobile
      • game-changing functionality: compute shaders, indirect drawing
      • other advanced featured
    • enable very rapid adoptation
      • expose hidden capabilities of current devicds
      • backward compatibility with es 2.0/3.0
      • run on any es 3.0 HW
    • improve application portability
      • tighter specs means less undefined behaviour
      • ever stricter conformance testing

ogl es 3.1 overview

  • bill qualcom
  • what’s new
    • compute shaders
      • tighly coupled graphics and GPU computing
    • draw-indirect
      • draw command parameters from memory instead of from args
    • SSO (separate)
  • compte shader basics
    • opencl, opengl
    • use gpu for general purpose computing
      • new single-stage pipeline, separate from the graphics pipeline
    • opg es
      • create a thread per vertex and a thread per fragment
    • glDispatch Compute()
    • SSBOs shader storage buffer objects
    • images
      • like texture, but writable
      • 4 image binidng points
      • image atomics avail as ext
    • serialziation and synchronization primitives
      • atomic counters
      • shader memory barriers
      • api memory barriers
  • draw-indirect
    • new way to dispatch work on gpu
      • draw call parameters stored in gpu memory
      • glDrawArraysindirect()
      • glDrawElementsindirect()
      • glDispatchComputeIndirect()
      • useful for draw parameters set by GPU
      • for example, in a compute shader
  • separate shader objects
    • mix and match vertex shaders and fragment shaders
  • Texture features
    • multisampled textures
    • stencil textures
    • textrue gather
  • shader language features
    • array of arrys
      • int a[2][3[4]
    • explict uniform location
      • layout(location=0) in vec4 position; // vertex shader
      • layout(location=2) out vec4 diffuse; //
    • shader bitfield ops
      • extract, insert, reverse, pack/unpack, count, lsb, msb
      • extended precision support
      • frexp / ldexp
    • shader helper invocation
      • bool gl_helperinvocation

ogles 3.0 ecosystem rollout

  • implementaions from every major mobile GPU vendor
    • tools, docs, sdks, etc
  • platform:
    • ios 7, iphone 5s >
    • android 4.3
  • unity mobile survey: 8% -> 21% adoptation
  • rollout es 3.1
    • doc, sdk, emulator, tools
    • imageination, nvida, qualcomm
  • platform support
    • adroid L
    • android extension pack, ASTC textures, etc
  • conformance test improvements
    • updated es 3.0 conformance relased in jan 2014
    • es 3.1 con is live 6 june 2014
    • 13 submissions from
      • arm, imagination, intel, nvidia, qualcomm, vivante
      • desktop: intel, nvidia
  • reference compiler
    • now supports virtually all of GLSL ES 3.1
    • compute shader
      • (minus arrays of arrays)
    • images
    • atomic counters
    • SSBOs
    • texture gather
    • texture LOD
    • SSO
    • .. and mores
test.comp

#version 310es
layout(localsize z= 1000) in;
void main()
{
 barrier();
 }

glslangValidator: test.comp
  • see -> khronos.org/ refernce compiler

basemark os I opengl es 3.1

  • rightware 2009 finland, www.rightware.com
  • biz area
    • ui creation tool, kanzi solutions
    • systems and chip performance evaluation software
  • ogles 3.1 test overview -> avil Q4 2014
  • demo
    • water using compute shader
    • particles 10000
    • post -processing
      • bloom, dof, anitializsing,e vignette
    • SSAD
    • deferred lighting G-buffer

google android extension pack (AEP)

  • google android extension pack (AEP)
    • android L mandates ogl es 3.1
  • aep es3.1 extensions
  • aep
    • tessellation
    • geometry shaders
    • ASTC texture compression + more
  • AAA grpahics effects
    • global illumination
    • physically-based shading
    • deferred rendering
    • HDR tone mapping
    • smoke and particle effects

GFXBench Car Chase Preview – OpenGL ES 3.1 + AEP

  • hdr deferred rendering pipeline
  • phycially-based shading
  • DOF and motion blur
  • 4k and astc support

GLES compute shader vs OpenCL

next gen opengl initiative

  • will talk a lot about why???
  • opengl is a huge success

  • problems with ogl today

    • gl programming model deosn’t match with modern gpu architeture
    • arcane and archanic syntax
      • legacy
      • needless complexity
    • cpu intensive
      • state validation, dependency tracking, error checking
      • hard to predict where cpu load will occur – varies with implementation
    • not multicore-cpu-friendly
      • primitive threading model, inconsisitenlty implemented
      • it’s broken right now
    • implementation variability
      • spec looseness
  • incremental change is not enough
  • goal on next gen
    • clean, modern architecture
    • multi-thread /multicore friendly
    • reduce cpu overhaead
    • architurecture-neutral
    • predictable performance
    • improved reliability and consistency
  • key design principle: explicit control

    • apllicaiton is telling the driver what it wants
  • status

    • Tom Olson -> Chair
    • Bill Licea-Kane(Qualcomm) -> Language chair
    • Graham Sellers(AMD) and Jeff Bolz(nvidia) -> API editor
  • who’s on board
    • sony, pixar, oculs, blizzard, epic games, ea, apple, unity, google, EA, RTT, amd, qualcomm
    • samsung, nvidia, imagination, , broadcom, arm, amd, hicorp, vivante, valve

OpenGL ecosystem support in Unity

  • democratizing games
  • demo game: ios “monument valley”
  • feature unity 5.0
    • deploy to webgl
  • unity multiplatform deployment
    [ios, android, windows, ps, web, xbox, wii, browsers, linux]
  • platform not using opengl backend
    => playstation, xbox,
  • driver support
    • www.g-truc.net
  • drivers and supports
    • nvidia 340.52
    • amd 14.7 rc1
    • intel 10.18.10.3652
    • mesa 10.2.2
    • macosx 10.9.4
  • opengl 4.2 is on most driver
  • reevaluation the opengl implementations
    • now it’s good
  • opengl in unity
    • extensions
      • gl_arb_es2_compatibiliyt
      • gl_arb_es3_compatibility
      • gl_arb_es3_1_compatibility
    • gles 4.1/ gl 4.3 or 4.2 + compute shader stuff
    • player base level-> es2.0
    • editor base level-> gl3.2
  • next ecosystem
    • 600K monthly ative users
    • 750K registered
    • 3 million registered developers
  • engine, support, learn, cloud, community, asset store
  • [glnext, ogl, dx, libgnm, mantle, metal]

3 speakers

  • Johan Andersson, TD, Frostbite – EA

Valve

  • John McDonald, Valve
  • gl is important
  • valve
    • valve did
      • steam for linux
      • steam runtime
      • steamos
      • VOGL -> debugger
    • valve contributed
      • glslang
      • LunarGLASS/ LunarGOO -> help opengl debugging
  • portal on android
    • uses ogles 2.x
  • grpahics today
    • direct3d is the primary on desktop
    • only on windows
    • tied to os ded10/11 are vista_
      • dx11 -> 87%
      • dx10.1 -> 15%
      • dx10 -> 13%
      • dx9 -> 4%
    • gl distribution
      • ogl 4.4 -> 67%
      • gl 3.3 -> 29%
      • gl 3.1 -> 4%x
  • glnext is shaping up
    • new and modern api

WEDNESDAY 1030..1200

  • global vr meetup
  • SVVR: http://svvr.com/

WEDNESDAY 1200..1300

  • Still Handling Very Large Data Sets – to Mars and Beyond, nasa, db&w finest plugins
  • http://www.jpl.nasa.gov/
  • Managing very large datasets, with an emphasis on planetary visualization, at the media lab at the Jet Propulsion Laboratory – a journey from the past to future.
  • We’ll be talking about the use of very large datasets at the JPL as well as large datasets in general. We’d love to see you joining us there.
  • bmw, JPL

XX

  • topics
    • history of tiled megatextures
    • current state of tiled / megatextures
    • tecnical challenges
    • future challenges
    • other datasets
  • basic principle of tiled-megatextures
    • mipmaps
    • different requirements
      • offline/ prouction rendering
      • realtime
  • offline / production rendering
    • pixar (surprise) textrue on demand
    • peachey 1990 PRMan 3.0 / tin toy et. al
  • achivement unlocked: tour de france 2000
  • starwars ep2
    • pod racing sequence (1999)
  • ideas spread a lot quicker
  • realtime
    • clipmapping -> ** term for mipmaps
    • terravision
    • keyhole(wow!) / google earth
    • quick diversion: legal minefields
    • games: IdTech 5 -> jon carmac used this
  • Idtech5/rage
    • one huge texture -> changed the way level artists are working
  • state of the art
    • common feature in production renderers
      • offten due to using the open source opeinimage io lib
      • some home-made solutions as well
    • mainstream in games
      • idtech5 and rage as a tech showcase
      • directly supported in state of the art apis
      • improved tools for texture artists
  • technical challenges
    • focus on production rendering
      • multiple cores
      • cache management
      • io management
      • io bandwidth
      • use in rendering (ie displacement)
      • suitability of image file formats -> exr, tif are fine..
  • future challenges
    • io and cpu power grows and rhe requirements
    • gpu rendering/computing
  • other large datasets
    • meshes
      • huge single meshees
      • repeated meshes
      • procedureal meshes
      • terrain, displacement
    • particles
      • KRAKATOA
    • large scale rendering – Manuka/Arnold
      • two renders has mesh compression of meshes
    • meshes
      • huge single meshes
      • in memory compression
      • load on demand, segment to mesh parts
      • spatting (for viewing of point clouds, qsplat)
      • volumetric solutions
    • repeated meshes
      • instancing
    • procedural meshes
      • “cheap” to recreate during the render or may be traced directly:
      • terrain
      • displacements
  • meshes: terrains
    • tesslate, optimize
  • meshes/point clouds: splatting
  • volumetric rendering
  • michael.wolf@db-w.com
  • ACTION ITEM
    • research huge texture technology used in rage 5 from idtech

Siggraph 2014: Day 3

AUG 12 2014, TUESDAY

TUESDAY 0945..1045

  • amd
  • mantle
    • new low level programming interface for windows pc
    • designed in collaboration with top game developers
    • lightweight driver that allows direct access to GPU
    • good for big game: battlefield 4
  • mantle benefits
    • higher graphics performance
    • empowering lower spec system
    • predictable performance behaviour
    • sharing optimizations between PCs and game consoles
    • new rendering techniques
      • more flexible, can be used on other gpu like intel and nvidia
  • why a new api?
    • success of the GCN architcture
      • uniform d
    • many collaboration with gaming industry and AMD
    • not a replacement for industry-standard API like DirectX or OpenGL
  • the small batch problem
    • draw calls are generated by the CPU to create work for the GPU
    • too many draw calls can make a system CPU-limited, leaving GPU under-utilized
    • batchig many objects into a single draw call adds code complexity and constrains artists
      • 3-5K draw calls
      • 10K draw calls
      • draw calls possible with heavy optimizations
      • 100K+ -> draw call target with mantle, 60fps
  • multi-threading
    • BEFORE
      • multi-threading gmae engines using directx and opengl
      • cpu0 -> game
      • cpu1 -> render
      • cpu2 -> dirver render
      • doesn’t scale beyond 2-3 cores
      • latency limits performance
      • driver thread often boecomes a bottleneck
    • AFTER
      • with mantle
      • cpu0 -> game
      • cpu1 -> render
      • cpu2 -> render
      • .. cpu5 -> render
      • scale linearly with number of cores
      • minimal latency
      • no driver thread overhead
  • MANTLE core features
    • app-controlled execution model
    • generalized resources
    • monolithic pipelines
    • fiexlible memory management
    • page talbe remapping
    • hybrid resource binding model
    • bindless -> huge in next
  • MANDTLE extended features
    • advanced flow control
    • advanced anti-aliasing functions
    • multi-device support
    • asynchronous compute
  • prog graphics application potential to use mantle
    • graphics api overhead
      • large/complex cad & DCC models consist of many individual assemblies, not easy or in many cases
      • AEC models of complex
      • Mantle reduces API overhead, avoiding inefficient state changes between those
    • multi-threading
      • CAD
    • memory management
      • cloud/vdi solutions may require more efficient use of graphics memory, partitioning, allocation,etc
      • mantle can provide
    • fast syncronization between compute eigne and graphic engines
    • drive optimizations
      • avoid driver bottlenects
      • dumb driver
    • data transfer control over PCIE
    • optimzations
      • optimize shader code to gain performance
      • use highly optimized code for specific
    • vertical market segments where mantle can help
      • cad, medical
    • to get the evaulation copy of firepro mantle driver
      • FirePro.Developers@amd.com
      • firepro developer sdk
      • http://developer.amd.com/tools-and-sdks/graphics-development/firepro-sdk
    • intel is approaching to them
  • ACTION ITEM
    • needs a slide
    • it’s a replacement of opengl or directx
    • will it be good? not sure

TUESDAY 1100..1200

  • maya
    • what’s new in maya 2015 ext 1
    • could import with dropbox a 123?
    • openubdiv surface
    • alembic
    • biforost -> prt
    • xgen
    • rendering
      • vp2.0
      • color management
      • maya hardware can be saved as exr
  • retopology options
    • auto and guided retopology – mudbox
    • cloud integration: reality capture data <- maya 2015 ext 1
    • gpu caching
    • data exchanging
    • maya quad draw -> manual retopology
  • MAYA 2015 new modeling toolkit
    • Mesh Editing Tools:
      • Target Weld
  • new boolean
    • will give you cut and uv will be there too
    • gemoetry + texture
  • new reduce tool
    • looks good, poly reduction
    • make nice clean geometry
    • some triangles, ngons quad -> click quad draw tool in modeling tools
      • hit magnet tool
      • and use quad draw
  • how to clean the dense scan model data

    • file -> send to unity, send to unreal
    • file -> game exporter
    • fbx review tool: free download
    • file -> cloud import/export => a360, 123D, dropbox
    • pipleline cache -> gpu ahe export -> exported to alembic

    • grab mesh

    • use magnet tool
    • quad draw tool
    • dragging and draw the quads
    • remeshing the surface
    • relax tool
    • soft seletion
    • sticking to the original surface
    • transfer map option
    • look at uv first
  • new uv editor
    • plannar mapping
    • grab edge of the head
    • cut and open up the head
    • fenecing
    • unfold3dunfold
    • unfold3d optimize
  • supports ptex file
  • ACTION ITEM
    • get the slide
    • trynew modeling tool
    • ptex
    • try export to alembic

TUESDAY 1230..0130

  • OSL open shading language from sony image works
  • more renderers and apps
  • released OSL 1.5
    • reparameter (enables lookdev tools)
    • shadergroup – specific attributes
      • per-group renderer output
      • re
  • featured change: group pickling
    • http://opensource.imageworks.com/?p=osl
  • send the param as a string
  • serilized a shader group
  • archive a shaer group, inluding .oso file
    • debugging
    • benchmarking/ optimization target
    • can be run with testshader or other tools – no renderer needed!
  • on deck
    • make it faster for SIMD-oriented renderers:
    • output cuda kernels somehow
    • optimized new usage cases
    • ecosystem

pixar

  • new version of renderman
  • renderman RIS & OSL wayne wooten
  • native ray trace renderer
  • whats’ RIS?:
    • integrators,
    • VCM bi-directional path tracer
    • BXDFs
      • glass, skin, disney
    • patterns
      • osl, SeExpr, Texture, Voronoise
  • SSS
  • Why RIS?
  • visual effects, 2014
    • simplfy workflow
    • fast feedback
    • easy to use
    • heavy gemoetry
    • volumes & particles
    • raytrace everything
  • typical day at work
    • check last night’s renders
    • dailies
    • breaks
    • meetings/ coffee
    • make it pretty(4h)
    • launch nightly build
    • 4-6 hours to get things done of your time
  • OSL
  • *** image from a wooden clock from carbon frame
    • oak feeding Disney BXDF
      • from Michel J. Anders
    • carbon fiber feeding Blend BXDF
      • from v0ray examples
    • backdrop 1 line simpletex.osl
    • OSL api is easy to use
  • OSL UDIM Texture
    • used in Mari
      • stands for U-dimension
      • UDIM(u,v) = 1000 + (10*u) + v + 1
  • OSL UDIM Texture
    • written in c++
  • OSL UDIM Texture
    • original naive implementation was very slow
      • vtune to the rescue
      • threading took a hit( multi-thread usting)
      • string manipulation was shlow
    • what to do?
      • bottleneck was OSL codebase
      • discussed with Larry Gritz
      • just under a week it ran fast
  • OSL UDIM Texture
    • open source really works
  • Evaluation Results
    • OSL pros
      • feels ‘homey’ to RSL users
      • targeted language and features
      • avoids plugin versionitis
    • OSL cons
      • disngle point, lack of SIMD/coherence
      • JIT time
      • LLVM targets libm.so,

OSL Gaffer

  • John Haddon, imageengine
  • What’s gaffer?
    • applicationf framework for VFX
    • node based
    • open source (BSD)
      • cortex,
  • two birds
    • gaffer -> lots of node, to manipulate scene hirerace
    • none for deforming each objs
    • need to allow
    • play with OSL
      • no renderer suppported it
  • loading shaders
    • search for shaders on env[“OSL_SHADER_PATHS”]
      • build menu
  • node graph
  • executing shaders
    • conver node network to OSL::shadingAttribState
    • implement renderservices::get_attributes() to provide access to prim vars and image channels
    • call shading system::execute() with fake shaderglobals
    • convert colsurecolor result ack to prim vars and image channels
    • abuse debug() closure
  • noise
  • show some demo
  • further work
    • UI implenetation
    • optimisation
    • using osl as a shading language
    • image convolution/ edge access
    • nei
  • http://github.com/imageengine/gaffer

#

  • general heckenberg? animal logic
  • physically based renderman pipeline => with renderman
  • very successful in 4-5 studios
  • “lego movie”
  • plastic -> lighting directions, specular lights
  • retraining the renderer translator
  • shader translation

TUESDAY 0200..0300

  • BOF, alembic
  • no major releases
  • moved to github/alembic/alembic
  • a little more collaboration
  • focus on stability, optimization
  • a few improvement, namespace support,
  • slow dev now.
  • will add hooks on importer and exporter
  • doulbe precision issue
  • game use abc becuase it contains more info than fbx
  • ACTION ITEM
    • check out github repo

TUESDAY 0300..0400

  • vray: sig: vray day1: Digital Domain | Rendering Heavy Assets: X-Men’s RFK Stadium
  • merged light cache
  • only 1 gi compuattion for the whole sequence
  • tree
    • 400 individual trees
    • billons of polygons
    • 42 species,
    • speed tree -> speedbridge => sent to DD pipeline
    • lookdev done in maya and 7vray
    • vray proxy -> full gemoetry -> vrayproxy
      • super fast scene handling
      • fast export and renders

Tehe RFK Stadium

  • lidar scan to sue as modeling reference
  • stadium size adjusted to fit over the whilehouse and match the live action
  • 40k vs 60k seats
  • main strcutre: concnreat
  • small details: lights, air con boxes, pipies and seats
  • split into 7 sections
  • shading and textures applied to what’s in camera only
    • keep the geo -> for gi calc
    • blew shading and texture
  • textures are from
  • lookdev done in maya & vray

destroying everything, stadium

  • stadium lifted by its metallic structure
  • use of our deferred render pipieline to render the seat
  • create a vrscene for a single seat -> one vrscene contains geo, mtl, tex
  • massive open file format
    • CDL rigging description
    • APF aniamtion info
  • render
    • load vrscene ->
  • landed stadium
    • destoyed stadium modelling from FX drop simulation
    • new faces created for the inside parts of the cube
    • no uvs on the new faces
    • first step ->
      • rgb mask color set in the vrayproxy accessed via vrayvertexcolor node
    • world space noise not “sticking” to the geometry
    • adding ‘rest’ attribute as another color set
    • noise mapped on the inside faces using rest color set as coordinate space
  • deep images
    • smoke coming out from houdini
    • all renders done with vray
    • breaking up was all done in houdini
  • vgo
    • tree -> proxy, geo proxy
  • render at 2K or 3K
    • using the external render time
    • amazon ec2?
    • rendering everything in house
    • render with vrimage and convert to exr
  • ACTION ITEM
    • should apply on the farm
    • one gi computation
    • vrayproxy
    • CDL, APF, vrscene

TUESDAY 0345..0445

  • Gaffer: vfx
  • kartick will go, i won’t

TUESDAY 0400..0500

  • renderfarming
  • file size -> use isilon
  • cloud -> use managed service and set up the bandwidth

Siggraph 2014: Day 2

AUG 11 2014, MONDAY

MONDAY 1000..1100

  • BOF First Steps in Large-Scale Rendering
    • [susan(pixar, white blonde), xx(), xx, xx, xx, david(pixar 19 years)] panel
    • pixar david, 19
    • ruca, from cube
    • ?? from singapore
    • **, toonbox
    • **, google, softare in pixar and animallogic
    • susan: pixar global rendering
    • Mark Hills
    • David Laur
    • mark.hills@framestore.com
  • big farm problem
  • selling softare for the small render farms
    • growing your farm
    • some caution
      • hetro geneous workers
      • HPC (high performance computing)
  • very often it’s linux because of the
  • multiple levels of caches in pixar render farm
    • go for memory upgrade
    • it’ll be better on IO node
  • using farm for computing, processing, simiulation, general purposes
    • 95% is rendering in pixar -> because of the GI
    • 5% is simulation, using all IO
  • segement the pool
  • remote rendering
    • route the low IO
    • desktop is high end gpus
    • linux =>
  • who’s doing GPU based the farm stuff
    • use GPU based farm
  • one frame per core
  • 4G per slot -> number of thread
  • bottle neck -> droppping
    • finding the sweat shop
  • killing the machine
    • keeping the tap on the production
  • multiple layers of controls
    • most of the queueing
    • multipile jobs coming in a smaller scope
  • 8 years ago, hope, multithreaded
    • the realizity -> it’s not scale
    • 24 core machines into 6-8 blocks
  • different versions and renders
    • 60% utilization=> cpu
    • threading control
    • 70-80%
  • always battle with cpu utilization with the vray settings
    • renderman, arnold
  • licensing ->
    • 1 license per machine -> tanking with jobs in one machine
  • rendering in the cloud
    • local farm busy
    • during the spike -> send to the remote farm
  • keep high io local
    • pick the high io render
  • how to pick high io???
  • data transfer to the remote is an issue..
    • make it fast to get it back and forth
    • latency between the managed render service ??
    • remote mounting from the
  • traffic is
    • syncing based file system
    • asset management is structured.
  • checking out in the cloud
    • 10Mbit -> typical studio
    • speed of the turn out
    • 1Gbit -> at teast for studio
    • i think we have 20G bit
  • pixar -> centralized in house *
  • small studio needs a turn key solution ???
    • provision them remote
  • sync or reupte
  • upload mananged servcies -> recommended??
    • upload the files through the web interface -> not good
    • too slow
    • managed service..don’t give you a lot of support, email that’s it
  • cost of the administration
  • configuration management software
  • multithreading on the virtual machines
  • provisioning the vm
    • get the core.
      • gareenteed the core, io -> provision
    • nfs over the network -> latency and slowness, cache on vps
  • is it a nfs, real solution??
  • segarate the traffic -> read from textrue, write to other file
  • traffic graph -> on the farm mount, know right away which file system is saturating the network
  • nfs
  • solution for the small company,
  • caching and latency
    • capital expance vs operation cost -> tax credit, decision where to go
    • financial advantage,, not only technical values
  • shortlist of priorities *
  • software tagging on the worker
  • scripts needed for rendring
    • how to wrap around?
    • send all render as an one big script?
  • wrapper script, lock
    • artists setup and the farm
  • design and the pipeline
  • opensource scheuler ??
    • 2 yeras ago -> liek 10
    • opendrag,,
  • good reference
    • best support of the vendors of the queuing software
  • amazon mka

MONDAY 0530..0700

  • threading
    • threading => intel tbb: use it all of them
    • openmp
    • c++ thread
    • boost threadin
  • problems
    • standarize tbb
  • pixar’s system => presto
  • the reality is
    • use tbb
  • multithread vfx
  • multhreading and vfx .com
    • get the contact information
  • using tbb ->
  • anyone using tbb
    • in gaming company, are using tbb on multiple gaming project
    • platform specific apis
    • internal framework wrapper
    • os differences
    • how do you finetuning betwen the different flatforms
      • follow the least constraints
      • what’s your targets? -> 4 cores?
      • job based, use core until the jobs done
      • until the stop is locked
      • game
      • do you know anything that will scale to?
      • 12 or 16 cores?
  • why not tbb?
    • openmp is easier
  • tbb, not good for length
    • microsft tpl -> we use opensource
  • c++ standard will have something similar
    • similar to c++ 0x
  • jame from intel
    • a few thoughts from intel’s perspective
    • true industry implementations
      • open cl, openmp port io,
    • standard is slow
      • tbb or current collections?
      • 2007 -> tbb, filling the gaps
    • lambda in c++11
  • gpu computing
    • cuda vs opencl
    • sebastian from nvida
    • 4 categories of reasons to use gpu computing
      • hardware: unified virtual memory. get the variables, fully debuggerable hardware.
      • tools do debug memory,
      • profiling
      • collection of libs
      • threast?, has cpu fall back
      • optics lib, raytracing on the gpu
    • what’s holiding up?
      • industry standards? -> monopoly
      • how serious
      • sweat spot?
    • highly unpredictable
    • why not use gpu?
    • unified memory -> mem and gpu ram
    • like to debug.
      • gpu = 100x of cpu
      • honest pair cpu vs gpu: 5 to 10x
  • rasment in disney animation studio
    • gpu 2x4x speed up
      • real optimzation on cpu
      • got 20 to 30x in the cpu
    • real problem not doing much of gpu
      • volitability of the environment, hardware and display driver
    • was painful 2-3 years on linux
    • design algorith that can scale
      • 10 of 1000s of threads
    • why not use cuda?
      • code maint cost
      • design first on cpu
    • weta digital
    • just optimizing and profiles
  • really goes down to standard platform
    • standard c++, opencl, tbb
  • cuda is getting mature
  • larger data set -> 28G ram
    • need a bigger memory
    • trying
  • 10G memory footprint on window.
  • gpu with 24G ram
  • why do we need gpu?
  • care about the energy efficiency?
    • data center cooling problmes
    • battery hot, cooling
  • performace and cooling?
  • power distribution in the data center
  • DOE, exso scale comuting
  • 1000 cores -> parallel programming
  • what goes inside of the cpu
  • energy efficient -> time efficient
    • avoid moving the data around
    • optimizing the data movement
    • main memory vs cpu or gpu
    • data movment within the chip, l1, l2 and registers
  • triple nested for loop to pass the asignment ->
    • intel xxx using what/
    • assignment from the code
  • intel vtune amp -> cpu and memory bandwith per socket
  • mpc : layer upon layer *
  • autodesk maya animation
    • 95%
    • using tbb
    • dreamworks session -> two way to improve
      • graph level parallism =>
      • node level parallism =>
    • motion builder -> make it past?
      • threading at the beginning
      • internal scheduler
      • thread classes
      • split and runnign as they want
    • story tool
      • not happy with the speed
      • optimize the speed
    • autodesk to dreamworks
      • dreamworks prima
      • built from scratch
      • 20 years old
      • thread safe
      • keeping it thread safe
    • whole lot of unit test
      • threading bug in the code
      • you can fix it
      • used tbb a lot -> hitting some limitations
    • intel new thing -> sbb(vecotrization)
      • check it out after siggraph

Siggraph 2014: Day 1

AUG 10 2014, SUNDAY

Sunday 1130..1230 nVidia see the big picture

  • largest flat wall screen
  • immersive displays
    • MOSAIC with sync
    • sync’s up to 4 gpus in workstations
    • visbox, cave, quadro K5000 + 6000
  • UHD USING CLUSTERS
    • Quadro Sync
    • 40 quadro sync cards in a cluster
    • control via NVAPI
    • control + monitor using NVWMI
    • BARCO/Elimit
  • not every surface is flat
    • WARP + INTENSITY Adjustment API
    • projection correction
    • curved surfaces
    • projection mapping
    • projection mappping on to a one fifth scale physical car
  • quadro features for high resolution display walls
    • MOSAIC
    • GPU direct 4
    • 4K at 60Hz
    • Warp + Intensity API
    • NVAPI/NVWMI – promatically control driver, remotely control remote gpu
  • From SD to 8K
    • exponontial Pixel Growth
  • from HD to 4K & Beyond
    • 1080p
    • UHD 4K – 3840×2160
    • DCI 4K – 4096×2160
    • DILA 4K – 4096×2400
  • Driving ultra high res displays
    • max single cable bandwithds
    • DP
      • version 1.2 upto 4k@60Hz
      • version 1.1a upto 4K@30Hz
    • HDMI
      • 2.0
    • DVI
  • drive multi-displays
    • independent desktops
  • with MOSAIC
    • one large desktop: 4K 16 monitors
  • MOSAIC supported on NVS + quadro
    • unified desktop up to 15 displayes
  • “mosaic with sync” features
    • seamless tear-free displays
    • sli bridge K6000, K5000
  • why is sync important?
  • multi-gpu sync
    • framelock
    • stereo lock
    • swap lock
  • vertical sync
  • framelock/genlock
  • stereo lock
  • swap buffers
    • taking longer than 1/60 sec
  • swap buffers in a cluster
    • each nodes with different fps(60/30/15/10 fps)
    • need a mechanism to ensure that each node will swap at the same time
  • nvidia opengl extension (via nvapi)
    • swap group and swap barrier
    • swap group -> provides synchronizaiton multiple gpus in a sigle host
    • swap barrier -> provides syncronization of gups accross multiple nodes
    • use rj45
  • what sync do
    • sync multiple display
    • coordinate buffer swaps
    • align the scan out of multpile display, gpus systems
    • maintain stereo alignment
    • sync to an internal or external timing
  • G-SYNC game monitors
    • different approach -> tells display when to update
    • instead of wait until vsync happens
  • MOSAIC
  • bezel and overlap correction
  • NV-WARP – WARP + INTENSITY API

SUNDAY 0200,,..0300

  • exchanging the material information
  • nvida mdl = material definition language
  • nvida ARC team
  • nvida iray
  • Iray Photoreal -> Iray interactive
  • nVidia UI composer
  • MDL usage example
    • iray photoreal
  • nvidia iray
    *
  • Iray 2014 Rendeing modes
    • realtime – fast, 15pfs ~ 60 fps
    • interactive – middle 20 fps ~ 2 fps
    • photoreal – best, 10fps ~ min
  • traditional shading language part
    • texturing => material definition => material implementation
  • new say
    • MDL
      • proedural programming language
      • declarative material definition
    • Renderer
      • rasterizer
      • raytracer
      • pathtracer
  • MDL is not a shading language
  • MDL
    • materail
      • surface: bsdf(scattering), emission(edf, emission) intensity
      • volume: vdf scattering, scattering_coefficent, abosrption_coeffcient
      • geometry: displacement, cutout_opacity, normal,
      • backface
      • ior, thin_walled
  • MDL: elemental distribution functions
    • BSDF(bidirectional scattering distribution functions)
    • EDF(emissive distribution functions)
    • VDF(Volumen distribution functions)
  • distribution function combiner
    • normalized mix, clamped mix, weighted layer
    • fresnel layer, custom curve layer, measured curve layer
  • measured materials
    • fast scan for believealbe material
    • quantitative measurements for predictive rendering
      • there is a device to measure the lights
  • AxF from X-Rite Spatially Varying (SV) BRDF
    • Analytic material model
    • measurement drives model parameters
      • paper: Practical SVBRDF capture in the frequency domain, siggraph 2013
  • measured isotropic BRDF
    • 3d data
    • can be used as any other SBDF in MDL
      • device to get light : radiant zemax imaging spheres device
  • Improving Dark Grazing Angles
  • procedural proramiming language
    • pure, side-effect free functions with read-only render state
      • texture lookup
      • texture placement
      • proedural textures
    • Iray 2013: Built-in functions provide a flexible instruction set for runtime execution
    • Iray 2014: fully programmable environment – clouded sky video
    • Iray 2015: full programmability – without losing GPU acceleration
  • C-like language
  • additionals benefits of MDL
    • physically-based materials are an easy-to-use paradigm
    • supports modern rendering algorithms
    • allows simple compilers and early optimizations
    • enables fast rendrers, especially on parallel architectures
    • GPU firendly
    • supports material catalogs
  • Practical layered material example
    • customer asked a “shader” to render realizstic cloth materials
      • color changes and complex
      • reference for requirement
      • “a practical microcylinder appearance model for cloth rendering”, sig 2013, iman sadeghi
      • usually this endes up complex, rendering engine dependent shader code
    • core ideas from the paper
      • if far, it’s not distinguishable
      • cloth has complex anisotropic behavior
      • threads are bent which creates secondary highlights
      • warp and weft might be of different olors, -> complex color variation.. 90 vertical, 0 horizontal
    • make it in MDL
      • warp -> anisotropic glossy BSDF
      • weft -> anisotropic
      • warp + weft -> complex reflective behavior of cloth
      • transluenccy
      • the full material
  • just by using mdl, building blocks
    • no more complex shader writing
    • good building blocks
  • MDL in commercial products
    • Iray for Maya
    • mental ray
    • Sketch up . bloom unit
    • realistyServer BIM IQ
    • UI composer
  • demo
    • nvidia Iray viewer
      • download
  • at the end of 2014
    • mentalray will support iray
  • GREAT SESSION
    • IT’S
  • supporting MDL
    • MDL specification is publicically available
      • MDL SDK => part of Iray SDK
      • contailns compiler and optimizer
    • Lightworks
      • nvidia’s partenr in Iray and MDL on your application
  • www.nvidia-arc.com => Iray => MDL
    • lightworks design booth #916

SUDNAY 0300..0430

  • BLENDER FOB, room 3 east hall
  • 5 mil downloades, 1% active user => 50,000 active users?
  • open source
    • ILM – openexr, alembic
    • pixar – opensubdiv
    • sony – open shading/color/image
    • disney – ptex
  • blender foundation
    • found in 2002 june
    • hires 5 devs for support work
    • handle donations and store incomes

MeCab을 이용한 한글 형태소 분석

한글 형태소 분석을 할려고 이것 저것 알아보다가 도달한 결론이 MeCab이란 일본에서 만든 형태소 분석기를 알게 되었다.

1. Ubuntu

1.1 설치

add-apt-repository ppa:tsuchiya/mecab 
apt-get install mecab 
apt-get install libcurl4-openssl-dev
mecab -v # 설치 잘 되었는지 확인하기 위해서

우분투 12에서 최신버전은 0.98 우분투 14.04L 에서는 0.996이 최신버전으로 설치되었다.

1.2 파일들

실행: /usr/bin/mecab /usr/lib/mecab => 다른 실행파일들 설정: /usr/local/etc/mecabrc 로그: 사전: /usr/share/mecab/dic/ipadic /usr/share/mecab/dic/juman

1.3 설정 파일

;
; Configuration file of MeCab
;
; $Id: mecabrc.in,v 1.3 2006/05/29 15:36:08 taku-ku Exp $;
;
;dicdir =  /usr/local/lib/mecab/dic/ipadic
;dicdir = /usr/share/mecab/dic/kdic/final
dicdir = /usr/local/lib/mecab/dic/mecab-ko-dic
; userdic = /home/foo/bar/user.dic

; output-format-type = wakati
; input-buffer-size = 8192

; node-format = %m\n
; bos-format = %S\n
; eos-format = EOS\n

/etc/mecabrc => 우분투에서 설정파일은 여깃다.

1.4 명령어

mecab
mecab -v 

2. Mac OS X

2.0 Update homebrew

brew update cd $(brew --prefix) git fetch origin git reset --hard origin/master brew update brew upgrade nginx brew --version

2.1 Installation

brew install mecab brew install mecab-ipadic

mecab -v # 설치 잘 되었나 확인하기 위해서

2.2 파일들

실행: /usr/local/bin/mecab symlinked to /usr/local/Cellar/mecab/0.994/bin/mecab 설정: /usr/local/etc/mecabrc 로그: 사전: /usr/local/Cellar/mecab/0.994/lib/mecab/dic/ipadic /usr/local/Cellar/mecab/0.994/lib/mecab/dic/kdic /usr/local/Cellar/mecab/0.994/lib/mecab/dic/

2.3 사전파일

kjune@eve /usr/local/Cellar/mecab/0.994/lib/mecab/dic $ unzip mecab-ko-dic.zip Archive: mecab-ko-dic.zip creating: mecab-ko-dic/ inflating: mecab-ko-dic/model.bin inflating: mecab-ko-dic/matrix.bin inflating: mecab-ko-dic/char.bin inflating: mecab-ko-dic/sys.dic inflating: mecab-ko-dic/unk.dic inflating: mecab-ko-dic/left-id.def inflating: mecab-ko-dic/right-id.def inflating: mecab-ko-dic/rewrite.def inflating: mecab-ko-dic/pos-id.def inflating: mecab-ko-dic/dicrc

3. ruby랑 엮기

그 다음에 루비랑 엮기위해서는 루비 바인딩이 필요한데 이를 위해서는 Natto라는 gem이 필요하다.

Gemfile에 natto 추가하고 bundle install해주면 된다.

gem 'natto'

4. 한글 사전

4.1 cogniti

한글 형태소 분석을 하려면 한글사전이 필요하다. 공개로 구할 수 있는 것은 cogniti님이 github에서 돌리신 mecab-kdic이 있다. 이거 가지고 해보자. 먼저 github에서 git clone해온다.

git clone http://github.com/cogniti/mecab-kdic.git

프로젝트 루트에서 make하면 사전파일이 만들어지는 구조이다.

개별적으로 환경이 맞지 않아서 수정해 줘야할 부분이 있는데, 내 경우에는 파일 2개 손대야 헸다. seed/Makefile하고 final/Makefile하고, 사전을 만들기 위해서는 mecab에서 제공하는 3개의 실행파일이 필요한데 그 경로를 잘 지정해줘야한다.

kjune@eve ~/_tmp/mecab-kdic $ cat seed/Makefile 
all:
  /usr/local/Cellar/mecab/0.994/libexec/mecab/mecab-dict-index -p -d . -c UTF-8 -t UTF-8 -f UTF-8
  /usr/local/Cellar/mecab/0.994/libexec/mecab/mecab-cost-train -c 1.0 corpus model
  /usr/local/Cellar/mecab/0.994/libexec/mecab/mecab-dict-gen -o ../final -m model

clean: rm -f *.dic *.bin model model.txt pos-id.def kdic.csv

Makefile은 이렇게 생겼다.

kjune@eve ~/_tmp/mecab-kdic $ cat final/Makefile 
all:
  /usr/local/Cellar/mecab/0.994/libexec/mecab/mecab-dict-index -d . -c UTF-8 -t UTF-8 -f UTF-8

clean:
  rm -f *.dic *.bin *.def *.csv dicrc model model.txt

설정 잘 맞추고 make잘 하면 final디렉토리에 파일들이 생긴다.

최종적으로 필요한 사전파일은 다음과 같다.

char.bin
dicrc
left-id.def
matrix.binc
pos-id.def
rewrite.def
right-id.def
sys.dic
unk.dic

맥의 경우에 사전파일들의 위치는 다음이므로.. kdic이라는 디랙토리를 하나 만들어주고 만들어진 바이너리들을 symlink로 연결해줫다.

/usr/local/Cellar/mecab/0.994/lib/mecab/dic/kdic

lrwxr-xr-x   1 kjune  admin   43 Dec 10 17:31 char.bin -> /Users/kjune/_tmp/mecab-kdic/final/char.bin
lrwxr-xr-x   1 kjune  admin   40 Dec 10 17:30 dicrc -> /Users/kjune/_tmp/mecab-kdic/final/dicrc
lrwxr-xr-x   1 kjune  admin   46 Dec 10 17:31 left-id.def -> /Users/kjune/_tmp/mecab-kdic/final/left-id.def
lrwxr-xr-x   1 kjune  admin   45 Dec 10 17:32 matrix.bin -> /Users/kjune/_tmp/mecab-kdic/final/matrix.bin
lrwxr-xr-x   1 kjune  admin   45 Dec 10 17:32 pos-id.def -> /Users/kjune/_tmp/mecab-kdic/final/pos-id.def
lrwxr-xr-x   1 kjune  admin   46 Dec 10 17:32 rewrite.def -> /Users/kjune/_tmp/mecab-kdic/final/rewrite.def
lrwxr-xr-x   1 kjune  admin   47 Dec 10 17:32 right-id.def -> /Users/kjune/_tmp/mecab-kdic/final/right-id.def
lrwxr-xr-x   1 kjune  admin   42 Dec 10 17:23 sys.dic -> /Users/kjune/_tmp/mecab-kdic/final/sys.dic
lrwxr-xr-x   1 kjune  admin   42 Dec 10 17:30 unk.dic -> /Users/kjune/_tmp/mecab-kdic/final/unk.dic

여러사전 중에서 어떤 사전을 선택할까는 mecabrc에서 정한다.

-rw-r--r--   1 kjune  admin  373 Dec 10 17:57 mecabrc
kjune@eve /usr/local/Cellar/mecab/0.994/etc $ cat mecabrc 
;
; Configuration file of MeCab
;
; $Id: mecabrc.in,v 1.3 2006/05/29 15:36:08 taku-ku Exp $;
;
;dicdir =  /usr/local/Cellar/mecab/0.994/lib/mecab/dic/ipadic
dicdir = /usr/local/Cellar/mecab/0.994/lib/mecab/dic/kdic

; userdic = /home/foo/bar/user.dic

; output-format-type = wakati
; input-buffer-size = 8192

; node-format = %m\n
; bos-format = %S\n
; eos-format = EOS\n

소스부터 다시 빌드하기

mecab-kdic 사전 만들다가 에러나서 소스로 부터 다시 빌드ㅎ하고 있다. 최신버전 0.994

sudo apt-get install mecab mecab-utils mecab-naist-jdic mecab-ipadic-utf8 mecab-jumandic-utf8 libmecab-java libmecab-jni python-mecab libmecab-ruby1.9.1

소스로 빌드하기 ubuntu 12.04 LT에서

  • download 0.994 http://code.google.com/p/furigana-injector/wiki/MecabInstallationMacOSX

    sudo aptitude install build-essential sudo aptitude install libc6-dev wget http://mecab.googlecode.com/files/mecab-0.994.tar.gz tar zxfv mecab-X.X.tar.gz cd mecab-X.X ./configure –with-charset=utf8 make make check su make install ldconfig

4.2 은전한닢 프로젝트

검색에 쓸만한 오픈소스 한국어 형태소 분석기 http://eunjeon.blogspot.ca/2013/02/blog-post.html

4.2.1 mecab 용 한글 사전 설치하기

1) 먼저 mecab-ko-dic 파일을 다운로드 받는다. https://bitbucket.org/eunjeon/mecab-ko-dic/downloads

다운로드 받은 소스를 풀어서 build한다.

wget https://bitbucket.org/eunjeon/mecab-ko-dic/downloads/mecab-ko-dic-1.6.1-20140814.tar.gz
tar zxfv mecab-ko-dic-XX.tar.gz
$ cd mecab-ko-dic-XX
$ ./autogen.sh
$ configure
$ make
$ su
# make install

문제점 => configure하는데서 에러가 났다.

checking for mecab-config... no
configure: error: mecab-config is not found in your system

해결책

apt-get install libmecab-config

다시 시작하니 설치가 잘 되었다. 마지막에는 다음과 같은 메시지가 나온다.

echo To enable dictionary, rewrite /etc/mecabrc as \"dicdir = /usr/lib/mecab/dic/mecab-ko-dic\"
To enable dictionary, rewrite /etc/mecabrc as "dicdir = /usr/lib/mecab/dic/mecab-ko-dic"
root@ubuntu:~/tmp/mecab-ko-dic-1.6.1-20140814# cd /usr/lib

마지막으로 make install을 하면 파일들이 다음 디렉토리로 설치된다.

root@ubuntu:/usr/lib/mecab/dic# cd mecab-ko-dic/
root@ubuntu:/usr/lib/mecab/dic/mecab-ko-dic# ll
total 120224
drwxr-xr-x 2 root root     4096 Feb 24 21:33 ./
drwxr-xr-x 3 root root     4096 Feb 24 21:33 ../
-rw-r--r-- 1 root root   262560 Feb 24 21:33 char.bin
-rw-r--r-- 1 root root     1421 Feb 24 21:33 dicrc
-rw-r--r-- 1 root root    81685 Feb 24 21:33 left-id.def
-rw-r--r-- 1 root root 20548480 Feb 24 21:33 matrix.bin
-rw-r--r-- 1 root root  9800884 Feb 24 21:33 model.bin
-rw-r--r-- 1 root root     1680 Feb 24 21:33 pos-id.def
-rw-r--r-- 1 root root     2540 Feb 24 21:33 rewrite.def
-rw-r--r-- 1 root root   121997 Feb 24 21:33 right-id.def
-rw-r--r-- 1 root root 92257020 Feb 24 21:33 sys.dic
-rw-r--r-- 1 root root     4196 Feb 24 21:33 unk.dic

이제 설치된 mecab에 맞춰서 mecabrc파일에 다음 내용을 추가한다.

/etc/mecabrc
dicdir = /usr/lib/mecab/dic/mecab-ko-dic

잘 설치되었나 확인하기

root@ubuntu:~# mecab
오늘은 날씨가 맑다.
오늘    NNG,*,T,오늘,*,*,*,*,*
은      JX,*,T,은,*,*,*,*,*
날씨    NNG,*,F,날씨,*,*,*,*,*
가      JKS,*,F,가,*,*,*,*,*
맑      VA,*,T,맑,*,*,*,*,*
다      EF,*,F,다,*,*,*,*,*
.       SF,*,*,*,*,*,*,*,*
EOS
^C

4.2.2 한글 사전용 mecab의 fork 설치하기

은전한닢 프로젝트에서 mecab을 fork한 버전을 배포하기 시작했다. 0.996 기반으로 한국어 특성에 맞는 기능을 추가했다고 한다.

wget https://bitbucket.org/eunjeon/mecab-ko/downloads/mecab-0.996-ko-0.9.1.tar.gz
tar xvfz mecab-0.996-ko-0.9.1.tar.gz
cd mecab-XX-ko-XX
./configure 
make
make check
su
make install

이러면 다음 파일들이 설치된다.

/usr/local/bin/mecab
/usr/local/bin/mecab-config
/usr/local/etc/mecabrc

이걸로 빌드를 하니까 사전 파일이 다른 디렉토리에 생긴다.

/usr/local/lib/mecab/dic/mecab-ko-dic

5. 결론

ubuntu랑 mac환경에서 mecab 설치하고 사전 설치하는 과정을 해봤는데, 현 시점에서는 은전한닢 프로젝트의 결과물을 가져와서 개발하는 것이 가장 안정적인 것으로 판단된다.

멀티 플랫폼 환경에서 ssh key 관리하기

Social coding의 시대에 맞춰서 대부분의 오픈 소스 프로젝트들과 소스컨트롤들이 git으로 가고 있는 현 추세에 맞춰서 ssh키 관리하는 글을 하나쓸까한다. 현재 필자는 윈도우 머신 2대, 맥, 리눅스와 호스팅서버, VPS 모두 사용해야하는 상황인데 그렇다고 해서 키를 시스템별로 다 새로 만들어서 사용하는 것은 아니기에 이렇게 가이드 문서를 하나 만든다.

먼저 ssh키를 하나 생성하도록 하자. ssh키는 자신의 이메일 주소당 하나 있다고 생각하는 것이 맞는 것 같다. 필자의 경우에는 개인용 프로젝트에 사용하는 이메일 계정에 맞는 키 1개 그리고 회사에서 일할 때 쓰는 키 1개 이렇게 2개로 운영하고 있다.

0. 환경 전제
윈도우에서는 cygwin을 사용한다고 전제하자. 맥과 리눅스는 터미널을 사용한다고 하자.

1. 키 생성하기
먼저 자신의 키를 하나 만들어야 한다. 새 환경 중 아무데서나 다음의 키를 이용해서 키를 하나 만든다.

ssh-keygen -t rsa -f mike3dk

이러면 passphrase를 물어보는데 비밀번호를 잘 넣어주면 된다.
그러면 자신의 홈디렉토리의 .ssh 디렉토리안에 다음의 2개의 파일이 만들어진것을 볼 수 있을 것이다.
~/.ssh/mike3dk
~/.ssh/mike3dk.pub
.pub파일은 자신의 퍼블릭키이고 그냥 파일은 자신의 프라이빗키이다. 공개키는 여기저기 막 놓아두어도 상관없지만 private키는 잘 간수하는 것이 좋겠다. Github나 다른 Git관련 솔루션에서 내 계정과 연결할 때 쓰는 공개키가 바로 mike3dk.pub 이 파일이다.

2. 이 키를 이용해서 ssh 로그인해보기
이 키를 이용해서 다른 사이트에 로그인을 해보자.
ssh myid@remotehost.com
그러면 ssh 비밀번호를 물어보는데 그 키를 넣어주면 된다.

3. 이제 비밀번호 없이 그 사이트에 로그인을 하는 법을 해보자.
크게 2가지 방법이 있는데 하나는..

cat .ssh/mike3dk.pub | ssh myid@remotehost.com 'cat >> ~/.ssh/authorized_keys'

다른 하나는 ssh-copy-id 명령어를 사용하는 법이다.

ssh-copy-id -i ~/.ssh/mike3dk.pub myid@remotehost.com

4. 이 키를 이용해서 다른 시스템에서 사용하기
방금 만든키를 이용해서 다른 환경에서 사용하기 위해서 먼저 이 키페어를 다른 시스템으로 복사를 한다.

scp ~/.ssh/mike3dk myid@mymac:~/.ssh/
scp ~/.ssh/mike3dk.pub myid@mymac:~/.ssh/

이 키를 이 시스템에서 쓰겠다고 하기 위해서는 먼저 ssh config파일을 만들어 줘야한다. 키가 설치가 안된 다른 환경으로 먼저 가서 config란 이름의 파일을 .ssh디렉토리 안에 만들고 다음의 내용을 추가해준다.

touch ~/.ssh/config
IdentityFile ~/.ssh/mike3dk

5. ssh-agent 띄우기

eval `ssh-agent`

6. key chain 사용하기
keychain은 자동으로 ssh-agent를 관리해주는 키 관리자이다.
맥에는 키체인이 GUI버전의 키체인이 기본으로 깔려있다.
대부분의 시스템에는 기본으로 설치되어있지는 않으니 일단 각 환경에서 설치하는 법을 적어보면

6.1 Ubuntu
1. keychain package를 인스톨한다.

sudo apt-get install keychain

2. keychain이 읽는 파일들을 만들어줘야 한다. 이를 위해 keychain을 한번 실행하도록 하자

mkang@W10023 ~ $ keychain

KeyChain 2.6.8; http://www.gentoo.org/proj/en/keychain/
Copyright 2002-2004 Gentoo Foundation; Distributed under the GPL

 * Initializing /home/mkang/.keychain/W10023-sh file...
 * Initializing /home/mkang/.keychain/W10023-csh file...
 * Initializing /home/mkang/.keychain/W10023-fish file...
 * Starting ssh-agent

3. .bashrc 파일을 편집해서 다음의 2줄을 넣어준다.

/usr/bin/keychain ~/.ssh/mike3dk
source ~/.keychain/$HOSTNAME-sh

그러면 ssh-agent를 한번에만 로딩하면서 키들은 캐쉬되어서 패스워드 안 넣어도 되는 시스템이 된다.

4. 현재 돌아가는 모든 ssh-agent를 죽일려면

keychain -k all

6.2 Cygiwn
1. keychain을 인스톨 한다.
cygwin 설치할 때 쓰는 setup.exe을 실행시켜 보면 util 쪽에 keychain 이 있다. 이거 설치하도록 한다.

2. 나머지는 위와 동일하다.

Python + PySide로 개발하기 위한 빌드환경 구축

최근에 크로스 플랫폼으로 UI를 가진 어플리케이션을 빨리 개발해야할 일이 생겨서 이것 저것 알아보다가 Python 2.7.3과 PySide 1.1.2로 개발하는 것이 좋겠다는 결론에 도달했다. 개발환경 구축을 위한 빌드오더는 다음과 같다.

Windows에서

1. 먼저 Python 사이트에서 다음 바이너리를 다운 받는다.
http://www.python.org/download/releases/2.7.3/
Windows x86 MSI Installer (32비트 버전이다)

2. 다운 받은 Python을 설치한다. 기본 설치디렉토리는 C:Python27이다.

3. PySide사이트에서 다음 바이너리를 받는다.
http://qt-project.org/wiki/PySide_Binaries_Windows
PySide-1.1.2-win32-py2.7.exe

4. 다운받은 PySide Binary를 설치한다. PySide가 C:Python27Libsite-packagesPySide 디렉토리에 설치된다.

5. 제대로 설치되었는지 확인하기 위해서 Python 명령창을 띄운다음 다음의 명령어를 준다.

import PySide
print PySide.__version__
1.1.2

Mac에서 (Mountain Lion)

맥에서는 기본으로 Python이 딸려오기 때문에 좀 불편하다. 먼저 이것을 제거하고 homebrew로 다시 깔아주자.

1. 운영체제에 딸려온 Python 제거하기

sudo rm -rf /Library/Frameworks/Python.framework/Versions/2.7
sudo rm -rf "/Applications/Python 2.7"
ls -l /usr/local/bin | grep '../Library/Frameworks/Python.framework/Versions/2.7' | awk '{print $9}' | xargs rm

2. Homebrew가 설치되었다는 가정하에, Homebrew를 이용해서 python이랑 pyside를 설치한다. Homebrew설치는 구글링해서 미리 해둘것

brew install python --framework --universal
brew install pyside
brew link python

3. 위에서와 마찬가지로 제대로 설치되었는지 확인하기 위해서 Python 명령창을 띄운다음 다음의 명령어를 준다.

import PySide
print PySide.__version__
1.1.2

새 터미널 git 환경설정

새로 계정이 생겨서 git작업을 할 때에 주로 추가하는 명령을 모아봤다.

git config --global alias.st status
git config --global alias.ci 'commit -a'
git config --global alias.co checkout
git config --global color.ui auto

일단 이것먼저 하고 시작한다.

우분투에 Nginx 웹서버 설치하기

우분투 12.04에 Nginx를 패키지 메니져를 이용해서 설치하는 법이다.
먼저 패키지 매니저를 업데이트 하고

apt-get update
apt-get upgrade --show-upgraded

nginx launchpad repository를 등록시키고

add-apt-repository ppa:nginx/stable

nginx를 설치한다.

apt-get install nginx

현재 설치된 버전을 확인하려면

kjune@mymachine ~/tmp $ nginx -v
nginx version: nginx/1.2.1