All posts by Eliezer Croitoru

squid-3.5, squid-4.0, squid-release

The Internet as a Talisman – SQUID 3.5.24 + 4.0.18 Released

The Internet as a Talisman
SQUID 3.5.24 + 4.0.18 Released

Some have more and others have less meaning for things in their lives and specifically for objects and to objectives. Most of the kids I have seen in my life have something embedded into them but not every eye can see the same things. Depends on the background and nature of the person he or she can see beyond the flesh and blood.
There is some part of it in the form of genetic material but I and many others believe it’s not the only thing. Every kid has it’s own embedded and unreplaceable soul. We have the option to show some reflection of a fraction from our soul to others either by plain text or by some Talisman, there is meaning in things. Even the most notorious researchers cannot deny that we all have some “meta” things embedded into us which the genome cannot touch. In a similar way to programming languages we can operate on the lower or the higher levels of this “meta” world.

“What will I choose to show for all in my piece of heaven?”
Is a question which everybody should ask themselves to my opinion.
Some decide to use their own body to show things in a form of a tattoo but it’s a much inferior form compared to many others. The basic level is with what exists already like the food you can serve: junk or nutritious. For some there is a conflict between these two choices but it’s a fact you can serve both non-junk and nutritious. A demonstration of what is considered honesty might not look the same as a shiny tattoo but it has much more warmth embedded into it. It will also affect much more than a single image or picture.

Today we have all kinds of tools to demonstrate an idea but still all of these cannot be compared to a living creature. I cannot pin point the exact difference but it’s there.

The Internet can be used as one of your own Talismans in your tiny piece of heaven. The issue with this compared to many more static and defined things is that there is an illusion that the Internet is composed of static things. For example “google will always be there” was a saying a friend of mine told me. The issue is that my tiny monitoring daemon shows from time to time that google and couple of google systems were down for a while. So the Internet is not that static. It has some “endpoint” which some humans operates.

When we open this tiny Router box that connects us to the Internet we will probably won’t find the exact reason for some site to be down. But, what box will we choose to put there at the edge of the house? What box will we choose to represent the Internet in our Heaven? Will it be a piece of junk which barely pumps the bits or will we use a top notch one?

It’s a simple question which we need to consider. For most of the users that question is being relayed to the sales man and he makes some of the choice much simpler for us. But, for the more technical(techy) person the answer is much more complicated. The simple person will want content filtering and OpenDNS or Symantec DNS systems will be good enough. In the other hand for the technical person which knows what’s inside the box, in some level the choice would be much more complicated. And it cannot really be relayed to the average sales man. Then the question might arise: “Maybe I will put together my own self compiled router and filtering system?” some might be able to make it past some level of the task, but eventually compared to a team of developers, what level of depth of this task will a single person mind would be able to surpass?
And to make some sense into the question: Would a single mind be able to compose a Talisman strong enough to be able to surpass the power of so many others out there on the Internet? Will a single person mind be able to make it so Internet surfing\browsing be smarter then viruses, malware, porn and many other things which most sane persons will not want to be affected by?

Indeed, it all starts with education but it also stats with a single box which exists now in so many homes, a PC that is connected to the Internet through a router. And do not think that I have some grudge against Facebook or YouTube or China but as much as I want to tell little kids how great is the experience of a marriage bond I would still block them from irrelevant details of this bond. It’s a part of a wisdom that I was granted  me freely without any form of payment.

Discretion is a wisdom and every Talisman has it’s own “Real” value, and yet maybe the sales man will tell you stories about the benefits about the product, but, eventually what you decide to buy is either the story of one sales man or another. This is since you cannot disconnect yourself from the World or the Internet. We need others which resides on the other side of this World Wide Web.

I had the chance to sit with some of the top notch developers in the content filtering world and when you ask them about their creations these do not even scratch their ideal solution. But yet so many enjoy their work so much. For the end-user, which doesn’t understand the complexity of the task, the “simple” box that the developer didn’t liked much might be the best choice of all.

Squid-Cache is one of these creations which are open to the public and can get criticized but yet to be replaced by other solutions for years. But why? Because it’s free? Because it’s Open-Source?
The answer is very simple: The developers behind the code.

The developers are the Brand!!
Like a chef they work on every release to prepare the next recipe of a Talisman which contains lots of goodness embedded into it.

Will it work for you?
There is an option that it will not fit your needs!!!

So what should I do?
Should I choose Squid-Cache or another product?
The answer to this question can be answered by a trial and error. Like a date, we put our efforts and try to see  the good in the other side of the contact.
Choose your Talisman: a 80’s compared to a 21’th century
I can only say that if you are looking for a ready to use off the shelf product Squid is not for you!
(Free Hints for research, try: ClearOS, NethServer, SmoothWall, ZeroShell, Zentyal, Untangle, Endian, pfSense, OpenSense )

Squid-Cache is one of a kind, a HardCore part of the web!
It’s the Software which will open your mind to a whole new HTTP world you have yet to see in the past.

All The Bests,
Eliezer Croitoru

References:

squid-3.5, squid-4.0, squid-release

When is the kid considered stable? – Squid 3.5.23+4.0.17

When is the kid considered stable?
Or
When is the software stable enough?

I am not developing software daily but it seems that the inner debate on the stability of our different “kids” and fruits of work is always there. Every time when I consider something as stable enough for production some other voice adds a doubt on the stability of a software.

Specifically, in the squid-cache and the open-source world the question is always on the board or the desktop. The source is open to everyone to find the next bug. Some are happy with what they have already while some expect the equivalent of a Ferrari.

When I am writing a software, I am doing my best effort to write it with the main goal that human lives can be entrusted to this software. One of the reasons why I am trying to meet such a goal is that in real life I try to do the same. When someone asks me a question or turns to me with a request or a word, I know that he asks for a reason. There is no chance in the world that the occasion is a result of only “A series of of unfortunate events”. I was asked in my past couple times “Why do you bother to answer?” and the answer is the simplest: the sanity of the other party is in my hands.

Sometimes a REDIRECT is the right answer but never DROP or REJECT. These are the actions of “war” and when we are talking about HTTP in general there is a war out there.  I had the choice in the past to work in couple layers of the Internet from the hardware to the application and I choose to invest lots of time on layer 4 and above.

There are many tools in this warzone and every time that a new tool is in use it get’s it’s own life cycle. The issue is that it takes time for every tool to become mature enough to serve different purposes.

Squid 3.5 is already in use by many users and admins for a while and is considered Stable for a very long time, but now aging start showing up.  There are different levels of maturity but the basic one is a period of 30 days uptime. We do expect more but a restart once every 30 days without any crash would be considered stable.

For now I am in the hunt for fatal bugs on the 3.5 series. The reason for this is to measure the maturity of the the branch!

Lately I have been working on couple tools and one of them is a library for distributed rate blacklists querying and blocking. Using this library I wrote an external acl helper for squid that can help many admins to use OpenDNS and Symantech or other internal DNS Blacklists.

The library sources can be found at:
https://github.com/elico/drbl-peer

And in binaries package at:
http://moodle.ngtech.co.il/drbl-extacl/

squid.conf example of usage:

external_acl_type dnsbl_check ipv4 concurrency=200 ttl=15 %DST %SRC %METHOD /opt/bin/squid-external-acl-helper -peers-filename=/opt/bin/peersfile.txt
acl dnsbl_check_acl external dnsbl_check
deny_info http://ngtech.co.il/block_page/?url=%u&domain=%H dnsbl_check_acl

http_access deny dnsbl_check_acl

Example of peersfile.txt:
http://moodle.ngtech.co.il/drbl-extacl/peersfile.txt

The syntax of the above file is:
type<space>address<space>path(for http services)<space>port<space>rate for the host(uint)<space>address which will indicate a blacklisted domain with spaces between them.
The type options are:

  • dns
  • dnsrbl
  • http
  • https

The http and https destinatiosn are queried with a HEAD request to the host and path and a match will be reflected in the response headers with “X-Vote” value to be “BLOCK”.
The dns and dnsrbl will be a match for one of the addresses which are defined from the six value on the definition and on ie matches in the line:

dns 208.67.220.123 / 53 128 146.112.61.104 146.112.61.105 146.112.61.106

would be the addresses:
146.112.61.104 or 146.112.61.105 or 146.112.61.106

and the weight of a match is 128 and specifically the default match weight of the helper is 128 and this line would be a match and no more lookups will be done. In the case which one list is not a match each of the listed would be tested until either found or the timeout(default 30 seconds) is reached. If the timeout or the list of peers was not found as fully matching the weight, the request will be allowed.

With this tool you can use regular DNS services on your system and on the proxy intercept the traffic and get a decision using a “consultation” with an external system not compromising your clients with special block pages that the dns redirect towards.

So we have both a proxy and a simple tool which can help us to prevent access to specific sites. The stability of Squid for this release is considered “Very Stable” but yet to be tested on a larger scale then 400 users. If you are managing a system which runs squid for filtering or caching that have more than 400 users please send us some input from the squid manager info page so we would be able to rate the state of the software.

I am planning to write a tiny tool\script that will help to scrap the squid manager info page and send the Squid-Cache systems some feedback. If you are a squid system administrator which are willing to share some statistics on your system with the project, please contact me at: eliezer@ngtech.co.il

I believe that if we could gather enough statistics we would be able to declare that the software passed the “masses” test compared to couple single systems.

All The Bests,
Eliezer Croitoru

On the plate:

References:

squid-3.5, squid-4.0, squid-release

The Squid “Persona”- Squid 3.5.21+ 4.0.14

Who Is The Squid Girl Persona?

squid_girl__shinryaku__ika_musume__minimalism_by_greenmapple17-d8u9mum

As a Squidder I know a girl or two but others are in the lookup for their Squid Girl Persona. Its’s not a simple task!!
To illustrate the idea: We as males used to the web with all sort of GET, POST, OPTIONS, PROFIND and all sort of normal methods.
But the Squid Girls is something else. She has brains!

I mean what would an empty shell be like? Would a Squidder would ever want a SHELL only system? Won’t some graphic or UI would be nice?
We do need some humen with enough Persona to take the SHELL up. As a friend told me once: We have great code, great kernel great compiler but.. with only one missing feature! The UI can be customized.
What can you say to someone like that? I mean, if the Software give you a full list of OPTIONS for the UI, what to choose from?

Despite to me preferring SHELL and scripting languages I believe that a good ICON and a GOOD Persona will give good taste for the whole cup.

Now let say we have a full stack proxy(AKA Router) and we can connect a network of Squids, what would be the next step?
Would it be plausible to “share” a cache_dir?
What about a ReadOnly cache_dir?
It sounds a bit weird but with good Cache Operators I believe that it would be possible to enhance a Squid-Cache network to serve a global Sharing system(compared to a CDN).

Google, Facebook, Yahoo MS, and couple others can be the main INPUT systems and the Squid-Cache Operators can push content into the right network at the right place and the right time.

The Squid Girl persona would be there just waiting for the Squidder and using a proxy, two, three, a BIG network two Squidders can be connected.

I am happy to say that from mu side of the picture Squid 4 feels stable to me and others needs to confirm that it work as expected for much larger systems.

Eliezer Croitoru

On the plate:

  • Squid full Tproxy LoadBalancer with The proxy protocol connection initiation.

Refrences:

squid-release

Faster is not always the answer!!

I am happy to publish the article for:
Squid-Cache 3.5.20 and 4.0.12 beta release.

The details about the the RPMs repository are at squid-wiki.
RPMs Available for CentOS, Oracle Linux, OpenSUSE Leap

Faster is not always the answer!!

When clients are not complaining?
What I mean is, did you ever seen a client complains about the speed of the Internet connection? No I do not mean that he or she complains it’s too slow but that it’s too fast?

I had the pleasure to meet couple clients which complained that the Computer is moving slow since their Internet connection speed was upgraded. No it wasn’t a joke and it is reality.

The scenario needs some background and context to sound a bit more realistic:
The client is in the age of about 80 and the PC is 2-3 year old. When the Internet connection was slow, the OS updates and AV P2P connections was slow. Every day the computer got shutdown around a specific hour and if was required some updates was applied. Now the issue is that since the Internet speed got faster, every couple hours an update from the AV was applied and almost every couple days an OS update was back on the table. The main issue was speed but with a twist “when I am disconnecting the router it’s working faster” he states.
Actually it took me quite a while to understand that a simple Desktop with about 4GB RAM should be enough to use: Skype, Word, Email and couple console based tiny pieces of software.

So why? why did the PC got slower?
I really do not know! It could be lots of IOPS that was dumped on a 5400 RPMs HDD or that the AV scanned the 2GB of updates repeatedly. I cannot answer what I never understood and from what I understood, faster is not always the good answer. However I can try to imaging that to verify that every signature of a file is still the same as it should be, might not be so easy for every PC.

These days I am counting the 10th month which my local testing Squid runs in a “full” http responses digest mode. Every single response was digested using the SHA256 hashing function and it feels like it’s not there at all. It’s not affecting my tiny 15Mbps line rate downloads  or my tiny servers farm.
Ho well it’s not the full and the whole truth!!

The full truth is that the users agreed to use the service in any form since they care more about their mind and soul rather then their comfort. They decided that they need some filtering system when they insert some data into their mind through their eyes. It’s as simple as it sounds. They know that their mind should be guarded under couple NAT systems and couple IDS+IPS since there are couple weird ideas out there on the Internet.

I am asking myself couple times every single day the questions like:

  • How do you want others to treat you when you have some need?
  • Would you want that others will do everything for you?
  • How would a “Plate Of Gold” look like?

And then my IDS+IPS system throws on me a big fat text exception with the header “We are humans, we need others!”.

And indeed this is an IDS+IPS which I didn’t built and every once in a  while I am asking myself, how many digest functions are in there?

  • CRC32
  • MD5
  • SHA1
  • SHA256
  • SHA512
  • SHA1024
  • SHA∞ ?

Is there an AES based one also in there?
And my answer is that I do not know what’s in there but I can see some “reflection” of something greater and better. Then I start to wonder, why all these clients wants their so well formed and solid and mature mind to be proxied using any solution? Would any human made solution ever match our genes?

I cannot give any “scientific” opinion but I can bring to the table things from others which have more weight then me on them either from life experience or scientific research. These do claim that the human genes are not “perfect” and there for there is always a need to “spice” the human mind and soul in order to allow it some level of progress. The most simple example of humans being affected is that kids tries to learn from their parents and later with time they try to learn from others. This state of learning curve can teach us that genes are not “everything”.

The answer will not always be “Faster” if you will get to the state of understanding and believing that it’s a rocket to your mind that’s hitting using words, pictures, tables, shapes and other things.

But!! don’t get paranoid!! Enough that you have another person in the house next to you and you are safe enough to not loose your mind. Enough that there is someone that can be asked directly or using a proxy and this world already feels much better then it was couple seconds ago.

All The Bests,
Eliezer Croitoru

Uncategorized

DNS as an API

I am happy to “Certify” Squid-Cache version 3.5.19 as
“Works For Me” on
CentOS(6+7), SLES(12SP1), Oracle Linux(6+7), RHEL(7), OpenSUSE(42.1 Leap), Debian(8.4), Ubuntu(14.04+16.04)

HTTP is commonly used as an API for many purposes in any industry and in many cases if you analyze an API specs and output you can see that some thinking was invested in it.

Around the Internet we can find many ideas about API’s and while some are well published others are long forgotten and are considered “old”. It is true that when you look at some of the API’s they might look “cryptic” or “malformed” but these have a purpose. Most of these APIs was meant to be public and as users we have access to all of them. But also many API’s requires some level of authentication or authorization which was clearly meant to not be fully public.

Some hackers around the world see the opportunity to “hack” something  when possible. From my own API’s which includes: HTTP, SMTP, DNS, WIFI HotSpot, Moblie and many others it is clear that some might think that it’s funny to send some malformed packets towards a Router or an AP. But I feel that there is a need to clear couple things out for any hacker.

Behind any System on the Internet there is some person which deserves respect. The fact that the API is there means that you are not allowed to hack it by it’s owner unless it was designed for it.
When comparing the real world to the Internet API’s not anyone can enter any door or any place. Not anyone can enter a closed party or a secured area. It would be a bit different since the minimum requirements to enter one place would not be the same for another.
For example, in the hackers world it’s known that there are ways to prove your value and earn your “nick” or “name”. Some hacking cultures are restrictive in their approach and respect any API avoiding the flame of war. While others think it’s better to hack some API as a Proof Of Concept or a Proof Of Knowledge.

White? Black? Green? Red? is there any meaning to all of these?
My answer is that all of these are hats, I do not have one and I do not want one. I am a simple person who has couple very simple API’s under his hands. But I learned that to give a good example is a profession. Specifically it’s not simple to give an example for a hacking kid. If any hacking kid wants to hack something, like in the real world, there are playgrounds for this sole purpose and an example would be canyouhack.it. Also these days if you want to learn how things work in the micro level we have Lots of free and open Virtualization platforms. These exist in any part of the Industry from the electricity level to the application.
All these tools was meant for the sole purpose of allowing the learning curve to be easy simple and safe, to use a real world power tool in an environment which will tolerate things which might not be acceptable in the real world API’s.

Not too far from the invention of HTTP the DNS system was invented and it’s an API like HTTP and many others. It is commonly used over UDP and has a very limited size and format but it has power in the same level as a button on a car dashboard. Technically it can and is being used in many places as a trigger to some system. Indeed UDP is not reliable at the same level of TCP but when the network equipment is trusted then there would be no reason to not use UDP.

A list of things that can be done using a DNS service messaging:

  • On\Off electrical switch
  • Identity signaling(AKA Port Knocking)
  • Banking transactions
  • Queue status updates
  • Alerts Signalling

And many other uses which can give an example to what an API can look like. I had the pleasure to read couple books about APIs published by Nordic APIs which gave me a fresh perspective on how others see an API and what might happen on the wild Internet that requires attention.

One key point which I learned from them is mentioned in the video “Good APIs aren´t built in a day”

And links to books from Nordic APIs  which I had the pleasure to read:

eBook Released: Securing the API Stronghold

API Security: The 4 Defenses of The API Stronghold

  • “Works For Me” means that it was tested on a testing environment under real world usage in a forward proxy mode with daily usage traffic such as Browsing News, Video, Learning and Games sites. Special applications that was tested are SKYPE, IRC and couple other applications inside a fully trusted network.
  • An Advice: Any system which sits against a non-trusted and a hostile public or private network should be “Harden” both in the squid configuration level and other lower levels.
  • This specific version(3.5.19) was tested also on Intercept proxy mode and ssl-bump but only on forward-proxy and not Intercept mode.
Uncategorized

A Proxy for each Internet user! The future!

What a proxy is as a tool? is it a war or a life assisting tool?

The Internet is a reflection of the real world, and the world in general can at times be a war-zone but is more of a heaven. A proxy basically is an assisting tool to the warrior of Internet. We can give it a shape of a Squid or of a Katana, but the tool by itself is there to help. And despite to the fact that in the science fiction and fantasy world the image of such a tool might be one, the truth is that it can take multiple forms. Also not every Internet warrior needs the same tools as another. Some needs raw Internet while others needs a more digested one, based on the age and experience.

Compared to the first human which god have created, we are engaging the world in a much higher level then raw basic input and output. And since we are at about the 6k year since this world creation, we have an embedded proxy in each and one of us. Every pair of parents shares with the kids some amount of tools. Yes this tool of war which helps us to digest raw Internet Input and Output.
A wise man once told me “Your tongue  has lots of power, do not do harm!” and I was wondering to myself couple years about this fact.
I knew that we have power in our words but compared to the raw hardware we are in a much higher level. We all have a proxy embedded inside of us and this is a fact. Now the question which stands in-front of  every Internet user and admin is whether he wants to utilize this tool as an assisting glass to the lower levels of the Internet and build the next and higher level, or to harm what is already there.

Little by little in life we discover our proxy powers and we can choose to either take these into our hands and to do good, or to use these powers in a way that will shame our form as a creation of a much better world. Yes, despite to what many non-experienced kids say we have a a very good foundation but we need to maintain it.

Uncategorized

Squid 3.5.15+16 release article

I am happy to release the article for the new squid 3.5.15+16
Until now the release was for the RPMs but I hope that from now on it will be much simpler to release every RPM.

The Fantasy Of Packaging
( A response to: Slamming Your Head Into Keyboard)

I was asked couple times “Why do you package?” and the simple answer is that I need it. But I must admit that it’s not because other developers and packagers doesn’t do a great job.
It’s just that I have couple specific systems that I operate and I need them to be up to date and until now I didn’t got the option to have someone that will take enough on his shoulders to make it happen in an Enterprise level.
And of-course it’s not that they do not want to, but since any Enterprise level distribution gives a good foundation for a very complex systems. When they develop and package they test and test and many other things until they can Insure themselves  and their clients.
It’s as simple as this. If in some case you have found an issue, they should help you with it by the level of contract you have with them.
For example since I have a Windows licensed PC and a licensed SLED Laptop I expect them to release at-least security updates.

I know that every major Linux Distribution does it’s job in a very good way. I know it since I can see their src.rpm packages and compare between my work to them.
Basically if you are a young man or a kid, the example from the scene at Toy Story which handy is opening the presents packages at the party and then he finds “Buzz light year” in a big box would describe to some how handling a package shouldn’t happen in the Enterprise level businesses.

(from about 09:00 to 14:00 minutes in the movie)

An admin should rely on the packages and packagers in general but he needs to plan and understand the nature of the update in order to anticipate what could be done in the update of the system.
If you have a big system which clients are using you cannot just run “yum update xzy” because you want to update. And specifically with squid you need a strategy that will allow you to sort of “bypass” this specific server and use another server in the mean while.
On small scale it’s simply adding another of squid instance on the same machine or to access squid using a haproxy LB.

It’s not really requires magic just basic skill knowledge and responsibility, if you are lacking one of these you should invest some time to acquire theses skills.

If for some reason for your or others business the generic packaging  doesn’t fit then you should understand that their aim was not you exactly but to another audience. And yes… it’s hard to see a package that does weird things like giving suid (chmod u+s XYX) to a file which actually is just a tiny script which should never run under root. But these things happens when the packager is novice enough to not understand they meaning of the work.

RPMs Automation: Here I come!!!

The squid-cache project and many others do not Package things in binary format and it’s important to understand that every Open-Source project probably have limited resources and goals and squid-cache is probably one of them.
All the sys-admins that uses my RPMs will probably enjoy them for a very long time but I have started to move from the manual hard labor RPMs creation to an automated one.
It means: That the packages will be pushed to the repository automatically for any squid release. My plan is to “certify” each of the RPMs or a release by a set of tests which are mainly manual rather then automated.
I haven’t decided yet what to do when a release fixed one bug but made another one but I will probably bump the RPMs version after basic testing.

And I would like to dedicate this amazing remix for all the amazing young IT industry mages that works many nights to give the Magical Internet be has great as it is now.

—www.youtube.com/watch?v=YFwoigs7Lhk
The above link was removed so another copy:

All The Bests,
Eliezer Croitoru