Categories
数据库

ORC字典实现

本文分析ORC中字典的实现,图1展示了本文中用到的列值序列。

When inserting

1 Building a dictionary by putting values into an in-memory array and encoding it with sequence order.

2 Store row values with encoding number(in-memory)

3 keep an index of distinct values with red-black-tree.

红黑树的节点是编码过的列值,但是排序时使用真实的列值排序,这样索引保证了列值是按真实值排序的。

当插入新数据的时候,需要判断字典中数据是否存在该数据,由于索引的存在查询复杂度是O(ln2)的,大大减小了查询时间,这是红黑树索引设计存在的意义。

2 When flushing column

分析:当数据插入完毕,flush数据的时候,有两项操作;第一按字典顺序重新编码;第二用新的字典对数据进行编码。这样做的目的是保证flush到磁盘的字典是按照真实值有序的,从而保证查询的高效。

1 Re-encoding data

2 Building a mapping between two encoding(in-memory)

3 when flush rows into disk, use new encoding number

4 column layout

3 When reading

字典使用的时候就比较简单了,我们先将问题抽象出来:用户发出scan的命令,此时ORC需要将列值(数字)Translate成列值,其实就是已知下标,然后读取数组的问题。

将关注点放到如何使用内存上,其实ORC的内存使用策略是将字典一次性读入内存。为什么能直接全量读入内存呢?首先字典的作用范围是一个Stripe,每个Stripe在构建的时候已经做过一次内存大小的限制了,所以全量读取一般也不会发生OOM。

版权声明:文章为作者辛勤劳动的成果,转载请注明作者与出处。

2,225 replies on “ORC字典实现”

We are a group of volunteers and opening a new scheme in our community.
Your web site provided us with valuable info to work on. You’ve
done a formidable job and our entire community will be grateful to you.

Today, I went to the beachfront with my kids.

I found a sea shell and gave it to my 4 year old daughter
and said “You can hear the ocean if you put this to your ear.”
She placed the shell to her ear and screamed. There was a hermit crab inside and it pinched her ear.
She never wants to go back! LoL I know this is
entirely off topic but I had to tell someone!

You are so cool! I do not suppose I have read anything like this before.
So great to discover somebody with some original thoughts on this issue.
Seriously.. many thanks for starting this up. This web site is something that
is required on the web, someone with some originality!

Hey there are using WordPress for your blog platform? I’m new to the blog world but I’m trying to get
started and create my own. Do you need any html coding expertise to make your
own blog? Any help would be really appreciated!

Pretty nice post. I just stumbled upon your weblog and wished to say
that I’ve really enjoyed surfing around your blog posts.
In any case I’ll be subscribing to your rss feed and
I hope you write again soon!

Simply want to say your article is as surprising.
The clarity in your post is simply spectacular and
i can assume you are an expert on this subject. Well with your permission allow me
to grab your feed to keep updated with forthcoming post.
Thanks a million and please continue the rewarding work.

Купить права на спецтехнику в компании on-prava можно всего лишь за несколько дней. Получение документа происходит на почте, поэтому права можно заказать из любого региона. Документ полностью официален и его можно проверить на сайте минсильхоза.
http://grantspass.chat/viewtopic.php?f=3&t=26207&p=168020#p168020

This is the right blog for anybody who hopes to understand this topic.
You realize so much its almost hard to argue with you (not that I actually would want to…HaHa).

You certainly put a brand new spin on a subject which has been written about for years.
Excellent stuff, just wonderful!

I am really impressed with your writing abilities as
neatly as with the structure to your blog. Is
this a paid subject or did you customize it yourself? Anyway keep up the excellent
high quality writing, it is uncommon to see a nice weblog like this one
today..

I like the valuable info you supply for your articles.
I’ll bookmark your blog and check once more right here frequently.
I am somewhat certain I will be told lots of new stuff proper right here!
Good luck for the following!

Hi there! I just wanted to ask if you ever have any problems with hackers?

My last blog (wordpress) was hacked and I ended up losing a
few months of hard work due to no backup. Do you have any
solutions to protect against hackers?

Do you have a spam problem on this site; I also am a blogger,
and I was wanting to know your situation; we have created some nice procedures
and we are looking to trade strategies with
others, be sure to shoot me an email if interested.

Hi there! I just wanted to ask if you ever have any issues with hackers?
My last blog (wordpress) was hacked and I ended up losing many months of hard work due to no back up.
Do you have any methods to protect against hackers?

Greetings from California! I’m bored to death at work so I
decided to check out your site on my iphone during lunch
break. I really like the information you present here
and can’t wait to take a look when I get home. I’m surprised at how fast
your blog loaded on my mobile .. I’m not even using WIFI, just 3G ..
Anyways, good blog!

Hello there! Quick question that’s completely off topic.
Do you know how to make your site mobile friendly? My web site looks weird
when viewing from my iphone. I’m trying to find a template or plugin that might be able to
correct this issue. If you have any recommendations, please
share. With thanks!

I was extremely pleased to find this website. I wanted to thank you for ones time for this particularly wonderful read!!

I definitely liked every part of it and I have you book marked to look at new things
in your blog.

It is perfect time to make a few plans for the longer term and it
is time to be happy. I’ve read this publish and if I
could I desire to counsel you some interesting
things or tips. Perhaps you can write next articles regarding
this article. I want to read even more issues about
it!

I’m really impressed with your writing skills as well as with the layout on your blog.
Is this a paid theme or did you customize it yourself?
Either way keep up the excellent quality writing, it is
rare to see a nice blog like this one these days.

Сумин поручил замам привлечь как можно больше федеральных денег. Сегодня губернатор Петр Сумин провел первое в наступившем году совещание со своими заместителями. Глава региона пожелал всем удачи, здоровья и успехов в работе в новом году. Петр Сумин подчеркнул, что 2009 год начался в условиях острой нехватки финансовых средств, он будет сложным, предстоит решать

This is the right site for everyone who would like
to find out about this topic. You realize so much its almost hard to
argue with you (not that I really will need to…HaHa). You definitely put a fresh spin on a topic that has
been discussed for ages. Great stuff, just great!

I was wondering if you ever thought of changing
the page layout of your site? Its very well written; I love what youve got to say.
But maybe you could a little more in the way of content so people could connect with it better.
Youve got an awful lot of text for only having one or
2 images. Maybe you could space it out better?

I think everything published was actually very reasonable.
But, think about this, what if you added a little content?
I am not suggesting your information is not solid, but suppose you added a title that
grabbed a person’s attention? I mean ORC字典实现 – Flying Pig is a little
boring. You should glance at Yahoo’s front page and see how they
create news titles to grab people to click. You might add a video or a picture or two to get
people excited about everything’ve got to say.
In my opinion, it could make your posts a little livelier.

A motivating discussion is definitely worth comment.
I believe that you ought to publish more on this subject matter,
it may not be a taboo matter but generally people do not speak about these topics.
To the next! Many thanks!!

hello!,I like your writing so a lot! share we communicate more about your post on AOL?
I need a specialist in this space to resolve my problem.
Maybe that’s you! Taking a look forward to see you.

Do you mind if I quote a few of your articles as long as I
provide credit and sources back to your webpage?
My blog is in the exact same area of interest as yours and my
users would really benefit from a lot of the information you provide here.
Please let me know if this okay with you. Cheers!

Hello! Do you know if they make any plugins to help with SEO?
I’m trying to get my blog to rank for some targeted
keywords but I’m not seeing very good success. If you know of
any please share. Kudos!

Hi there! I just wanted to ask if you ever have any issues with hackers?
My last blog (wordpress) was hacked and I ended up losing several weeks of hard work due to no data backup.
Do you have any methods to protect against hackers?

Hi, I think your site might be having browser
compatibility issues. When I look at your website in Safari, it looks fine but when opening in Internet Explorer,
it has some overlapping. I just wanted to give you a quick heads
up! Other then that, superb blog!

I used to be recommended this blog by my cousin. I am not sure whether or
not this submit is written by means of him as nobody else know such
distinctive approximately my problem. You are amazing!
Thanks!

excellent publish, very informative. I ponder why the opposite experts of
this sector don’t notice this. You must proceed your writing.
I am sure, you’ve a great readers’ base already!

Howdy! This post could not be written any better!
Reading this post reminds me of my good old room mate!
He always kept talking about this. I will forward this article to him.
Fairly certain he will have a good read. Thanks for sharing!

Thanks for the marvelous posting! I genuinely enjoyed
reading it, you’re a great author.I will remember to bookmark your blog and will often come
back from now on. I want to encourage one to continue
your great posts, have a nice weekend!

Just want to say your article is as astonishing.

The clearness on your post is simply great and i can suppose you’re a professional in this subject.
Fine along with your permission allow me to snatch your feed to stay
up to date with forthcoming post. Thanks 1,000,000 and please continue the
rewarding work.

Nice blog here! Also your website loads up fast!
What web host are you using? Can I get your affiliate link to
your host? I wish my website loaded up as fast as yours lol

My brother recommended I would possibly like this blog.
He was totally right. This publish truly made my day.
You cann’t believe just how much time I had spent for this info!
Thanks!

Simply wish to say your article is as astounding. The clarity in your post is just
excellent and i could assume you’re an expert on this
subject. Fine with your permission let me to grab your feed to keep up to date with forthcoming
post. Thanks a million and please continue the enjoyable work.

Woah! I’m really loving the template/theme of this website.
It’s simple, yet effective. A lot of times it’s tough to get that “perfect balance” between usability and appearance.

I must say that you’ve done a superb job with this.
Also, the blog loads super quick for me on Opera. Superb Blog!

Greate post. Keep writing such kind of info on your
page. Im really impressed by your site.
Hi there, You have done an incredible job. I will definitely digg it and in my opinion suggest to my friends.
I am confident they will be benefited from this website.

I loved as much as you’ll receive carried out right here. The sketch is tasteful, your authored material stylish. nonetheless, you command get got an shakiness over that you wish be delivering the following. unwell unquestionably come more formerly again since exactly the same nearly very often inside case you shield this increase.

This is really fascinating, You are an overly professional
blogger. I have joined your rss feed and stay up for searching for more of your
excellent post. Also, I’ve shared your web site in my
social networks

I think this is one of the so much important information for me.

And i’m satisfied reading your article. But wanna
observation on few basic issues, The web site taste is ideal, the articles is truly great : D.
Just right process, cheers

https://malinka64.ru/
Если вы хотите получить профессиональный эротический массаж – то нет ничего проще. Закрывайте глаза, думайте о чем – то приятном, а наши специалисты помогут вам окунуться в мир нежных эротических снов, откуда не хочется возвращаться.

Hi! I understand this is kind of off-topic but I had to ask.
Does managing a well-established blog such as yours take a large amount of work?
I am completely new to writing a blog however I do write in my journal daily.
I’d like to start a blog so I can share my experience and views online.
Please let me know if you have any ideas or tips for brand
new aspiring bloggers. Appreciate it!

When I initially commented I appear to have clicked the -Notify me when new comments
are added- checkbox and from now on whenever a comment is added I get four emails with the exact same comment.

Perhaps there is a way you are able to remove me from that service?
Kudos!

Hey just wanted to give you a brief heads up and let you know
a few of the images aren’t loading correctly.

I’m not sure why but I think its a linking issue. I’ve
tried it in two different internet browsers and both show
the same results.

Neat blog! Is your theme custom made or did you download it from somewhere?

A theme like yours with a few simple adjustements would really make my blog stand out.
Please let me know where you got your design. Appreciate it

We are a group of volunteers and starting a new scheme in our community.

Your web site provided us with valuable info to work on. You have done an impressive job and our whole community
will be thankful to you.

Hi there! I could have sworn I’ve visited your blog before but after going through some of
the articles I realized it’s new to me. Anyways, I’m definitely happy I stumbled upon it and I’ll be
book-marking it and checking back often!

Have you ever thought about writing an e-book or guest
authoring on other blogs? I have a blog based on the same topics you discuss and
would love to have you share some stories/information.
I know my viewers would enjoy your work. If you’re even remotely interested, feel free to send me an e-mail.

Howdy! This blog post couldn’t be written much better!
Reading through this post reminds me of my previous roommate!
He constantly kept preaching about this. I am going to send this post to him.
Pretty sure he will have a great read. Thank you for sharing!

Hello There. I found your blog using msn. This is an extremely well written article.
I will make sure to bookmark it and return to read more of your useful info.
Thanks for the post. I’ll definitely comeback.

I don’t even understand how I ended up right here,
but I thought this publish was great. I do not recognise who you are but certainly you’re going to a famous blogger in case you are not already.
Cheers!

My developer is trying to convince me to move to .net from PHP.
I have always disliked the idea because of the expenses. But he’s tryiong none the less.
I’ve been using WordPress on a number of websites for about
a year and am concerned about switching to another
platform. I have heard good things about blogengine.net.

Is there a way I can import all my wordpress posts into it?

Any kind of help would be greatly appreciated!

Oh my goodness! Impressive article dude! Many thanks, However
I am going through difficulties with your RSS. I don’t understand why I cannot
subscribe to it. Is there anybody getting the same RSS problems?
Anybody who knows the answer can you kindly respond? Thanx!!

I’m not that much of a internet reader to be honest but your blogs really nice,
keep it up! I’ll go ahead and bookmark your
website to come back down the road. Many thanks